2023-12-08T09:49:47.132 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-08T09:49:47.132 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419 --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --name yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/orig.config.yaml 2023-12-08T09:49:47.182 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-08T09:49:47.262 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419 branch: wip-yuri5-testing-2023-12-07-0740-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7483419' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2023-12-07-0740-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2023-12-07-0740-pacific sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1847 sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2023-12-07-0740-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD2mTZcolWLO2JNlYBeDpOo4vqvQ8kcgIwCIB0zlJ5HgDhlAsVAgKJaqSAuAdFR5tSdOxXPulHbdV4ejODiGtvg= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDp4zu2Wjlz0Wu3gTPx5rr7Z/Nhh+9F19811qM/0lEvQ51PG9q+ys4Im0cOWf2L6KOMXlfZpeXo3dFSN1s2kBGo= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-08_04:25:47 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-08T09:49:47.262 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa; will attempt to use it 2023-12-08T09:49:47.263 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks 2023-12-08T09:49:47.263 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-08T09:49:47.264 INFO:teuthology.task.internal:Checking packages... 2023-12-08T09:49:47.288 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '451ceae11b5d4caee05e94372fe658621eefc0b1' 2023-12-08T09:49:47.289 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-08T09:49:47.289 INFO:teuthology.packaging:ref: None 2023-12-08T09:49:47.289 INFO:teuthology.packaging:tag: None 2023-12-08T09:49:47.289 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-08T09:49:47.289 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T09:49:47.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-08T09:49:47.514 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-365.g451ceae1 2023-12-08T09:49:47.514 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-08T09:49:47.521 INFO:teuthology.task.internal:no buildpackages task found 2023-12-08T09:49:47.521 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-08T09:49:47.532 INFO:teuthology.task.internal:Saving configuration 2023-12-08T09:49:47.552 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-08T09:49:47.557 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-08T09:49:47.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-08 09:43:33.060496', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD2mTZcolWLO2JNlYBeDpOo4vqvQ8kcgIwCIB0zlJ5HgDhlAsVAgKJaqSAuAdFR5tSdOxXPulHbdV4ejODiGtvg='} 2023-12-08T09:49:47.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-08 09:43:33.061947', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDp4zu2Wjlz0Wu3gTPx5rr7Z/Nhh+9F19811qM/0lEvQ51PG9q+ys4Im0cOWf2L6KOMXlfZpeXo3dFSN1s2kBGo='} 2023-12-08T09:49:47.604 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-08T09:49:47.611 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-08T09:49:47.611 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-08T09:49:47.611 INFO:teuthology.run_tasks:Running task console_log... 2023-12-08T09:49:47.688 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff104c2da60>, signals=[15]) 2023-12-08T09:49:47.688 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-08T09:49:47.694 INFO:teuthology.task.internal:Opening connections... 2023-12-08T09:49:47.694 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2023-12-08T09:49:47.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:49:47.764 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2023-12-08T09:49:47.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:49:47.833 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-08T09:49:47.839 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2023-12-08T09:49:47.861 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2023-12-08T09:49:47.862 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2023-12-08T09:49:47.923 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-08T09:49:47.924 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-08T09:49:47.924 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-08T09:49:47.924 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2023-12-08T09:49:47.951 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2023-12-08T09:49:47.973 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-12-08T09:49:47.974 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2023-12-08T09:49:48.036 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2023-12-08T09:49:48.036 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8" 2023-12-08T09:49:48.036 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2023-12-08T09:49:48.036 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-08T09:49:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-08T09:49:48.037 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-12-08T09:49:48.059 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-08T09:49:48.068 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-08T09:49:48.073 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-08T09:49:48.073 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2023-12-08T09:49:48.076 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2023-12-08T09:49:48.096 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-08T09:49:48.103 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-08T09:49:48.103 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2023-12-08T09:49:48.138 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2023-12-08T09:49:48.162 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-08T09:49:48.218 INFO:teuthology.run_tasks:Running task kernel... 2023-12-08T09:49:48.234 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-08T09:49:48.234 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-08T09:49:48.234 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-08T09:49:48.235 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-08T09:49:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T09:49:48.255 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-12-08T09:49:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T09:49:48.256 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2023-12-08T09:49:48.315 INFO:teuthology.orchestra.run.smithi170.stdout:4.18.0-489.el8.x86_64 2023-12-08T09:49:48.316 INFO:teuthology.task.kernel:Running kernel on smithi170: 4.18.0-489.el8.x86_64 2023-12-08T09:49:48.316 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2023-12-08T09:49:48.317 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-489.el8.x86_64 2023-12-08T09:49:48.317 INFO:teuthology.task.kernel:Running kernel on smithi023: 4.18.0-489.el8.x86_64 2023-12-08T09:49:48.317 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2023-12-08T09:49:51.877 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - AppStream 32 MB/s | 34 MB 00:01 2023-12-08T09:49:52.405 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - AppStream 23 MB/s | 34 MB 00:01 2023-12-08T09:50:01.953 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 55 MB 00:03 2023-12-08T09:50:02.313 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 55 MB 00:03 2023-12-08T09:50:10.735 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - Extras 37 kB/s | 18 kB 00:00 2023-12-08T09:50:11.336 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-12-08T09:50:11.612 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - Extras 12 kB/s | 18 kB 00:01 2023-12-08T09:50:11.646 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-12-08T09:50:12.196 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-12-08T09:50:12.490 INFO:teuthology.orchestra.run.smithi170.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-12-08T09:50:12.596 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-12-08T09:50:14.068 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-12-08T09:50:16.671 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 399 kB/s | 24 kB 00:00 2023-12-08T09:50:17.882 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 404 kB/s | 24 kB 00:00 2023-12-08T09:50:20.588 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-08T09:50:20.588 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-08T09:50:20.876 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T09:50:20.877 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T09:50:20.878 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T09:50:20.878 INFO:teuthology.orchestra.run.smithi023.stdout:Install 3 Packages 2023-12-08T09:50:20.878 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T09:50:20.878 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 90 M 2023-12-08T09:50:20.878 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 96 M 2023-12-08T09:50:20.878 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T09:50:21.481 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-08T09:50:21.481 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-08T09:50:21.769 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T09:50:21.770 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T09:50:21.770 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T09:50:21.770 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout:Install 3 Packages 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T09:50:21.771 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 90 M 2023-12-08T09:50:21.772 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 96 M 2023-12-08T09:50:21.772 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T09:50:22.639 INFO:teuthology.orchestra.run.smithi023.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 6.8 MB/s | 10 MB 00:01 2023-12-08T09:50:24.623 INFO:teuthology.orchestra.run.smithi023.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 10 MB/s | 36 MB 00:03 2023-12-08T09:50:25.190 INFO:teuthology.orchestra.run.smithi023.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 11 MB/s | 43 MB 00:04 2023-12-08T09:50:25.190 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T09:50:25.190 INFO:teuthology.orchestra.run.smithi023.stdout:Total 21 MB/s | 90 MB 00:04 2023-12-08T09:50:25.900 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T09:50:26.352 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T09:50:26.353 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T09:50:26.399 INFO:teuthology.orchestra.run.smithi170.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 2.2 MB/s | 10 MB 00:04 2023-12-08T09:50:27.644 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T09:50:27.663 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T09:50:28.316 INFO:teuthology.orchestra.run.smithi170.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 5.6 MB/s | 36 MB 00:06 2023-12-08T09:50:29.024 INFO:teuthology.orchestra.run.smithi170.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 6.0 MB/s | 43 MB 00:07 2023-12-08T09:50:29.024 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T09:50:29.025 INFO:teuthology.orchestra.run.smithi170.stdout:Total 12 MB/s | 90 MB 00:07 2023-12-08T09:50:29.707 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T09:50:30.131 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T09:50:30.131 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T09:50:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T09:50:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-08T09:50:31.369 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T09:50:31.387 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T09:50:34.161 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T09:50:34.786 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-08T09:50:34.946 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-08T09:50:35.329 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-08T09:50:38.315 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-08T09:50:38.694 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-08T09:50:39.002 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-08T09:50:39.255 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:50:42.128 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-08T09:50:42.417 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:02.466 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:02.491 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:02.854 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-08T09:51:02.855 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-08T09:51:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:03.845 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T09:51:03.846 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T09:51:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-08T09:51:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-08T09:51:03.846 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T09:51:03.846 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T09:51:04.009 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2023-12-08T09:51:04.236 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-08T09:51:04.236 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-08T09:51:04.236 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-08T09:51:04.236 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-08T09:51:04.236 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-08T09:51:04.236 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-08T09:51:04.236 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2023-12-08T09:51:04.237 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2023-12-08T09:51:04.384 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:04.410 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:04.456 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-08T09:51:04.456 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-08T09:51:04.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.02 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 129.el8 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Group : System Environment/Base 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 9265586 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-08T09:51:04.491 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:Relocations : (not relocatable) 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : CentOS BuildSystem 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T09:51:04.492 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2023-12-08T09:51:04.493 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-08T09:51:04.493 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 4.18.0-526.el8.x86_64 2023-12-08T09:51:04.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-08T09:51:04.751 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:04.751 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-08T09:51:04.751 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-08T09:51:05.524 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-08T09:51:05.524 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T09:51:05.524 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T09:51:05.524 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-08T09:51:05.524 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-08T09:51:05.524 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T09:51:05.525 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T09:51:05.666 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-12-08T09:51:05.831 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2023-12-08T09:51:05.920 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-08T09:51:05.920 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-08T09:51:05.920 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-08T09:51:05.920 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-08T09:51:05.920 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-08T09:51:05.921 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-08T09:51:05.921 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2023-12-08T09:51:05.921 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2023-12-08T09:51:06.131 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-08T09:51:06.131 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-08T09:51:06.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2023-12-08T09:51:06.165 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.02 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 129.el8 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Group : System Environment/Base 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 9265586 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-08T09:51:06.166 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:Relocations : (not relocatable) 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : CentOS BuildSystem 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T09:51:06.167 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2023-12-08T09:51:06.168 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-08T09:51:06.168 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 4.18.0-526.el8.x86_64 2023-12-08T09:51:06.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-08T09:51:07.893 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2023-12-08T09:51:08.004 INFO:teuthology.orchestra.run.smithi023.stderr:done 2023-12-08T09:51:08.005 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2023-12-08T09:51:08.035 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.akAaY8XZf5 2023-12-08T09:51:08.036 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.akAaY8XZf5 2023-12-08T09:51:08.108 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.akAaY8XZf5 2023-12-08T09:51:08.249 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.akAaY8XZf5 is 6KB 2023-12-08T09:51:08.298 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.akAaY8XZf5 2023-12-08T09:51:08.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2023-12-08T09:51:08.396 INFO:teuthology.orchestra.run.smithi023.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-08T09:51:08.396 INFO:teuthology.orchestra.run.smithi023.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-08T09:51:08.396 INFO:teuthology.orchestra.run.smithi023.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-08T09:51:08.396 INFO:teuthology.orchestra.run.smithi023.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-08T09:51:08.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-08T09:51:08.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2023-12-08T09:51:10.128 INFO:teuthology.orchestra.run.smithi170.stderr:done 2023-12-08T09:51:10.135 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2023-12-08T09:51:10.163 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.lJJWpYDDVx 2023-12-08T09:51:10.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lJJWpYDDVx 2023-12-08T09:51:10.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.lJJWpYDDVx 2023-12-08T09:51:10.387 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.lJJWpYDDVx is 6KB 2023-12-08T09:51:10.435 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.lJJWpYDDVx 2023-12-08T09:51:10.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2023-12-08T09:51:10.518 INFO:teuthology.orchestra.run.smithi170.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-08T09:51:10.518 INFO:teuthology.orchestra.run.smithi170.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-08T09:51:10.518 INFO:teuthology.orchestra.run.smithi170.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-08T09:51:10.519 INFO:teuthology.orchestra.run.smithi170.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-08T09:51:10.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-08T09:51:10.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2023-12-08T09:51:38.582 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-08T09:51:38.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T09:51:38.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:51:40.674 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-08T09:51:40.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-08T09:51:40.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:52:13.155 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-12-08T09:52:22.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T09:52:22.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:52:25.218 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-12-08T09:52:37.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T09:52:37.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:52:40.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-12-08T09:52:40.677 DEBUG:teuthology.orchestra.remote:timed out 2023-12-08T09:52:49.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-08T09:52:49.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:52:50.105 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-12-08T09:52:50.344 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-08T09:52:50.344 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-526.el8.x86_64"... 2023-12-08T09:52:50.344 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2023-12-08T09:52:50.361 INFO:teuthology.orchestra.run.smithi170.stdout:4.18.0-526.el8.x86_64 2023-12-08T09:52:50.361 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-08T09:52:50.361 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-08T09:52:50.361 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-08T09:52:51.362 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-08T09:52:51.362 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-08T09:52:51.428 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2023-12-08T09:52:51.447 DEBUG:teuthology.parallel:result is None 2023-12-08T09:52:55.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T09:52:55.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:53:13.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2023-12-08T09:53:31.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T09:53:31.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T09:53:32.989 DEBUG:teuthology.orchestra.run.smithi023:> true 2023-12-08T09:53:33.323 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T09:53:33.323 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-526.el8.x86_64"... 2023-12-08T09:53:33.323 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2023-12-08T09:53:33.341 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-526.el8.x86_64 2023-12-08T09:53:33.341 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-08T09:53:33.341 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-08T09:53:33.342 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-08T09:53:34.343 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-08T09:53:34.343 DEBUG:teuthology.orchestra.run.smithi023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-08T09:53:34.422 INFO:teuthology.orchestra.run.smithi023.stdout:ttyS1 2023-12-08T09:53:34.435 DEBUG:teuthology.parallel:result is None 2023-12-08T09:53:34.435 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-08T09:53:34.442 INFO:teuthology.task.internal:Creating test directory... 2023-12-08T09:53:34.443 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-08T09:53:34.446 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-08T09:53:34.470 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-08T09:53:34.477 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-08T09:53:34.487 INFO:teuthology.task.internal:Creating archive directory... 2023-12-08T09:53:34.487 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-08T09:53:34.506 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-08T09:53:34.545 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-08T09:53:34.570 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-08T09:53:34.570 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-08T09:53:34.573 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-08T09:53:34.610 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T09:53:34.622 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T09:53:34.625 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T09:53:34.636 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-08T09:53:34.637 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-08T09:53:34.645 INFO:teuthology.task.internal:Configuring sudo... 2023-12-08T09:53:34.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-08T09:53:34.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-08T09:53:34.695 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-08T09:53:34.705 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-08T09:53:34.705 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-08T09:53:34.735 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-08T09:53:34.755 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-08T09:53:34.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-08T09:53:34.898 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-08T09:53:34.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-08T09:53:34.989 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T09:53:34.990 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-08T09:53:35.058 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-08T09:53:35.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-08T09:53:35.160 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-08T09:53:35.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-08T09:53:35.289 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T09:53:35.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-08T09:53:35.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2023-12-08T09:53:35.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2023-12-08T09:53:35.419 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-08T09:53:35.453 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-08T09:53:35.894 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-08T09:53:35.901 INFO:teuthology.task.internal:Starting timer... 2023-12-08T09:53:35.902 INFO:teuthology.run_tasks:Running task pcp... 2023-12-08T09:53:35.911 INFO:teuthology.run_tasks:Running task selinux... 2023-12-08T09:53:35.919 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-08T09:53:35.920 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2023-12-08T09:53:36.004 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2023-12-08T09:53:36.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2023-12-08T09:53:36.082 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2023-12-08T09:53:36.084 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-08T09:53:36.084 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2023-12-08T09:53:36.118 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2023-12-08T09:53:36.119 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2023-12-08T09:53:36.161 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2023-12-08T09:53:36.161 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2023-12-08T09:53:36.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-08T09:53:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T09:53:36.191 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-08T09:53:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T09:53:36.230 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-08T09:53:36.231 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2023-12-08T09:53:36.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2023-12-08T09:53:36.297 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-08T09:53:36.307 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-08T09:53:36.313 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-08T09:53:36.466 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-08T09:53:36.483 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-08T09:53:36.484 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-08T10:01:13.429 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2023-12-08T10:01:13.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T10:01:13.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T10:01:13.503 DEBUG:teuthology.orchestra.run.smithi023:> true 2023-12-08T10:01:13.578 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2023-12-08T10:01:13.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-08T10:01:13.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-08T10:01:13.649 DEBUG:teuthology.orchestra.run.smithi170:> true 2023-12-08T10:01:13.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-12-08T10:01:13.716 INFO:teuthology.run_tasks:Running task clock... 2023-12-08T10:01:13.728 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-08T10:01:13.729 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-08T10:01:13.729 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T10:01:13.731 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-08T10:01:13.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T10:01:13.762 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-08T10:01:13.777 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-08T10:01:13.786 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-08T10:01:13.800 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-08T10:01:13.850 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2023-12-08T10:01:13.861 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2023-12-08T10:01:13.876 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-08T10:01:13.880 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2023-12-08T10:01:13.889 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-08T10:01:13.892 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2023-12-08T10:01:13.905 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-08T10:01:13.919 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-08T10:01:13.962 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2023-12-08T10:01:13.965 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-08T10:01:13.965 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2023-12-08T10:01:13.966 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.966 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.966 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.966 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.987 INFO:teuthology.orchestra.run.smithi170.stderr:bash: ntpq: command not found 2023-12-08T10:01:13.990 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-08T10:01:13.990 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-12-08T10:01:13.990 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.990 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.991 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.991 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:01:13.991 INFO:teuthology.run_tasks:Running task pexec... 2023-12-08T10:01:14.001 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-08T10:01:14.002 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi023.front.sepia.ceph.com 2023-12-08T10:01:14.002 DEBUG:teuthology.orchestra.run.smithi023:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-08T10:01:14.002 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi170.front.sepia.ceph.com 2023-12-08T10:01:14.003 DEBUG:teuthology.orchestra.run.smithi170:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-08T10:01:14.911 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:46 ago on Fri 08 Dec 2023 09:59:28 AM UTC. 2023-12-08T10:01:14.922 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:36 ago on Fri 08 Dec 2023 09:59:38 AM UTC. 2023-12-08T10:01:17.057 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:01:17.058 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:01:17.058 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:01:17.058 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:01:17.058 INFO:teuthology.orchestra.run.smithi170.stdout:Resetting modules: 2023-12-08T10:01:17.059 INFO:teuthology.orchestra.run.smithi170.stdout: container-tools 2023-12-08T10:01:17.059 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:17.059 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:01:17.059 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:01:17.059 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:17.218 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:01:17.219 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout:Resetting modules: 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:01:17.220 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:17.303 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:01:17.474 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:01:18.241 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:50 ago on Fri 08 Dec 2023 09:59:28 AM UTC. 2023-12-08T10:01:18.444 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:40 ago on Fri 08 Dec 2023 09:59:38 AM UTC. 2023-12-08T10:01:20.344 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:01:20.527 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:01:20.527 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:01:20.527 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:01:20.527 INFO:teuthology.orchestra.run.smithi170.stdout:Installing group/module packages: 2023-12-08T10:01:20.527 INFO:teuthology.orchestra.run.smithi170.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-08T10:01:20.527 INFO:teuthology.orchestra.run.smithi170.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-08T10:01:20.528 INFO:teuthology.orchestra.run.smithi170.stdout:Installing module profiles: 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout: container-tools/common 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout:Enabling module streams: 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout: container-tools rhel8 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2023-12-08T10:01:20.529 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:20.530 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 27 M 2023-12-08T10:01:20.530 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 89 M 2023-12-08T10:01:20.530 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:01:20.703 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:01:20.803 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 3.2 MB/s | 750 kB 00:00 2023-12-08T10:01:20.844 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 910 kB/s | 38 kB 00:00 2023-12-08T10:01:20.905 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:01:20.905 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:01:20.905 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:01:20.905 INFO:teuthology.orchestra.run.smithi023.stdout:Installing group/module packages: 2023-12-08T10:01:20.905 INFO:teuthology.orchestra.run.smithi023.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-08T10:01:20.906 INFO:teuthology.orchestra.run.smithi023.stdout:Installing module profiles: 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools/common 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools rhel8 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout:Install 13 Packages 2023-12-08T10:01:20.907 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:20.908 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 27 M 2023-12-08T10:01:20.908 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 89 M 2023-12-08T10:01:20.908 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:01:20.911 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.0 MB/s | 1.0 MB 00:00 2023-12-08T10:01:20.964 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.0 MB/s | 241 kB 00:00 2023-12-08T10:01:21.014 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.5 MB/s | 178 kB 00:00 2023-12-08T10:01:21.056 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 591 kB/s | 25 kB 00:00 2023-12-08T10:01:21.115 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2023-12-08T10:01:21.375 INFO:teuthology.orchestra.run.smithi023.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.7 MB/s | 750 kB 00:00 2023-12-08T10:01:21.425 INFO:teuthology.orchestra.run.smithi023.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 759 kB/s | 38 kB 00:00 2023-12-08T10:01:21.557 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 6.1 MB/s | 3.9 MB 00:00 2023-12-08T10:01:21.600 INFO:teuthology.orchestra.run.smithi023.stdout:(3/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.0 MB/s | 1.0 MB 00:00 2023-12-08T10:01:21.626 INFO:teuthology.orchestra.run.smithi023.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.2 MB/s | 241 kB 00:00 2023-12-08T10:01:21.701 INFO:teuthology.orchestra.run.smithi023.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.3 MB/s | 178 kB 00:00 2023-12-08T10:01:21.743 INFO:teuthology.orchestra.run.smithi023.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 597 kB/s | 25 kB 00:00 2023-12-08T10:01:21.802 INFO:teuthology.orchestra.run.smithi023.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2023-12-08T10:01:21.999 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 5.7 MB/s | 2.5 MB 00:00 2023-12-08T10:01:22.049 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.0 MB/s | 53 kB 00:00 2023-12-08T10:01:22.091 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 983 kB/s | 41 kB 00:00 2023-12-08T10:01:22.341 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 5.4 MB/s | 9.6 MB 00:01 2023-12-08T10:01:22.377 INFO:teuthology.orchestra.run.smithi023.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 5.0 MB/s | 3.9 MB 00:00 2023-12-08T10:01:22.450 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 6.1 MB/s | 8.1 MB 00:01 2023-12-08T10:01:22.450 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:01:22.450 INFO:teuthology.orchestra.run.smithi170.stdout:Total 14 MB/s | 27 MB 00:01 2023-12-08T10:01:22.686 INFO:teuthology.orchestra.run.smithi023.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 8.2 MB/s | 2.5 MB 00:00 2023-12-08T10:01:22.716 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:01:22.736 INFO:teuthology.orchestra.run.smithi023.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 1.0 MB/s | 53 kB 00:00 2023-12-08T10:01:22.737 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:01:22.737 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:01:22.794 INFO:teuthology.orchestra.run.smithi023.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2023-12-08T10:01:22.912 INFO:teuthology.orchestra.run.smithi023.stdout:(12/13): buildah-1.32.0-1.module_el8+712+4cd1bd 5.3 MB/s | 9.6 MB 00:01 2023-12-08T10:01:22.997 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:01:23.000 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:01:23.036 INFO:teuthology.orchestra.run.smithi023.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 6.6 MB/s | 8.1 MB 00:01 2023-12-08T10:01:23.037 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:01:23.037 INFO:teuthology.orchestra.run.smithi023.stdout:Total 12 MB/s | 27 MB 00:02 2023-12-08T10:01:23.312 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:01:23.334 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:01:23.335 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:01:23.609 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:01:23.612 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:01:23.665 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:01:23.874 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-08T10:01:24.036 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-08T10:01:24.244 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-08T10:01:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:01:24.483 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-08T10:01:24.522 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-08T10:01:24.708 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-08T10:01:24.901 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-08T10:01:25.134 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-08T10:01:25.146 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-08T10:01:25.746 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-08T10:01:26.129 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-08T10:01:26.352 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-08T10:01:26.531 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-08T10:01:26.774 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-08T10:01:26.785 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-08T10:01:27.010 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-08T10:01:27.148 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-08T10:01:27.216 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-08T10:01:27.462 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-08T10:01:27.866 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-08T10:01:28.035 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-08T10:01:28.249 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-08T10:01:28.418 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-08T10:01:28.825 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-08T10:01:28.837 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-08T10:01:28.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-08T10:01:28.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-08T10:01:28.837 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-08T10:01:28.838 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-08T10:01:29.047 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-08T10:01:29.249 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-08T10:01:29.655 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-08T10:01:29.655 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:29.655 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:01:29.655 INFO:teuthology.orchestra.run.smithi170.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:29.655 INFO:teuthology.orchestra.run.smithi170.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:29.655 INFO:teuthology.orchestra.run.smithi170.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:01:29.656 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-08T10:01:29.706 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-08T10:01:29.707 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-08T10:01:29.707 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-08T10:01:29.707 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-08T10:01:29.707 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-08T10:01:29.707 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-08T10:01:29.707 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-08T10:01:30.245 DEBUG:teuthology.parallel:result is None 2023-12-08T10:01:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-08T10:01:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:30.551 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:01:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:01:30.552 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:01:31.460 DEBUG:teuthology.parallel:result is None 2023-12-08T10:01:31.461 INFO:teuthology.run_tasks:Running task pexec... 2023-12-08T10:01:31.472 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-08T10:01:31.472 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi023.front.sepia.ceph.com 2023-12-08T10:01:31.472 DEBUG:teuthology.orchestra.run.smithi023:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-08T10:01:31.473 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi170.front.sepia.ceph.com 2023-12-08T10:01:31.473 DEBUG:teuthology.orchestra.run.smithi170:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-08T10:01:31.638 DEBUG:teuthology.parallel:result is None 2023-12-08T10:01:31.664 DEBUG:teuthology.parallel:result is None 2023-12-08T10:01:31.664 INFO:teuthology.run_tasks:Running task install... 2023-12-08T10:01:31.673 DEBUG:teuthology.task.install:project ceph 2023-12-08T10:01:31.674 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}} 2023-12-08T10:01:31.674 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'} 2023-12-08T10:01:31.674 INFO:teuthology.task.install:Using flavor: default 2023-12-08T10:01:31.680 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-08T10:01:31.680 INFO:teuthology.task.install:extra packages: [] 2023-12-08T10:01:31.681 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-08T10:01:31.681 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-08T10:01:31.681 INFO:teuthology.packaging:ref: None 2023-12-08T10:01:31.681 INFO:teuthology.packaging:tag: v16.2.4 2023-12-08T10:01:31.682 INFO:teuthology.packaging:branch: None 2023-12-08T10:01:31.682 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T10:01:31.889 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:01:31.889 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:01:31.892 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-08T10:01:31.892 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-08T10:01:31.892 INFO:teuthology.packaging:ref: None 2023-12-08T10:01:31.892 INFO:teuthology.packaging:tag: v16.2.4 2023-12-08T10:01:31.892 INFO:teuthology.packaging:branch: None 2023-12-08T10:01:31.892 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T10:01:32.100 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:01:32.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:01:32.319 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-08T10:01:32.320 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-08T10:01:32.336 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-08T10:01:32.336 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-08T10:01:32.360 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-08T10:01:32.360 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:01:32.360 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-08T10:01:32.368 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-08T10:01:32.369 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:01:32.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-08T10:01:32.395 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-08T10:01:32.395 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-08T10:01:32.395 INFO:teuthology.packaging:ref: None 2023-12-08T10:01:32.395 INFO:teuthology.packaging:tag: v16.2.4 2023-12-08T10:01:32.395 INFO:teuthology.packaging:branch: None 2023-12-08T10:01:32.395 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T10:01:32.396 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-08T10:01:32.401 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-08T10:01:32.402 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-08T10:01:32.402 INFO:teuthology.packaging:ref: None 2023-12-08T10:01:32.402 INFO:teuthology.packaging:tag: v16.2.4 2023-12-08T10:01:32.402 INFO:teuthology.packaging:branch: None 2023-12-08T10:01:32.402 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T10:01:32.402 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-08T10:01:32.468 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-08T10:01:32.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-08T10:01:32.551 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-08T10:01:32.555 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-08T10:01:32.621 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2023-12-08T10:01:32.622 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2023-12-08T10:01:32.624 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2023-12-08T10:01:32.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2023-12-08T10:01:32.892 INFO:teuthology.orchestra.run.smithi170.stdout:76 files removed 2023-12-08T10:01:32.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2023-12-08T10:01:32.924 INFO:teuthology.orchestra.run.smithi023.stdout:76 files removed 2023-12-08T10:01:32.953 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw 2023-12-08T10:01:33.393 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 1.2 MB/s | 74 kB 00:00 2023-12-08T10:01:33.494 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 239 kB/s | 15 kB 00:00 2023-12-08T10:01:33.581 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 369 kB/s | 74 kB 00:00 2023-12-08T10:01:33.594 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-12-08T10:01:33.726 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 145 kB/s | 15 kB 00:00 2023-12-08T10:01:33.832 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 23 kB/s | 1.7 kB 00:00 2023-12-08T10:01:34.878 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - AppStream 27 MB/s | 34 MB 00:01 2023-12-08T10:01:35.133 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - AppStream 27 MB/s | 34 MB 00:01 2023-12-08T10:01:44.179 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 55 MB 00:03 2023-12-08T10:01:46.650 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 55 MB 00:04 2023-12-08T10:01:52.529 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2023-12-08T10:01:53.212 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-12-08T10:01:54.005 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.6 MB 00:00 2023-12-08T10:01:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - Extras 88 kB/s | 18 kB 00:00 2023-12-08T10:01:55.482 INFO:teuthology.orchestra.run.smithi170.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-12-08T10:01:55.575 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2023-12-08T10:01:56.735 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 8 - PowerTools 6.7 MB/s | 6.6 MB 00:00 2023-12-08T10:01:57.944 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 6.9 MB/s | 16 MB 00:02 2023-12-08T10:01:58.278 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-08T10:01:59.862 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-12-08T10:02:01.874 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 444 kB/s | 24 kB 00:00 2023-12-08T10:02:03.904 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 428 kB/s | 24 kB 00:00 2023-12-08T10:02:06.451 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:02:06.453 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-08T10:02:06.454 INFO:teuthology.orchestra.run.smithi170.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-08T10:02:06.455 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout:Install 25 Packages 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2023-12-08T10:02:06.456 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:06.457 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 59 M 2023-12-08T10:02:06.457 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:02:07.385 INFO:teuthology.orchestra.run.smithi170.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 34 MB/s | 6.0 MB 00:00 2023-12-08T10:02:07.418 INFO:teuthology.orchestra.run.smithi170.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 747 kB/s | 25 kB 00:00 2023-12-08T10:02:07.452 INFO:teuthology.orchestra.run.smithi170.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2023-12-08T10:02:07.494 INFO:teuthology.orchestra.run.smithi170.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 12 MB/s | 486 kB 00:00 2023-12-08T10:02:07.762 INFO:teuthology.orchestra.run.smithi170.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 23 MB/s | 13 MB 00:00 2023-12-08T10:02:07.836 INFO:teuthology.orchestra.run.smithi170.stdout:(6/27): librgw2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2023-12-08T10:02:07.862 INFO:teuthology.orchestra.run.smithi170.stdout:(7/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 447 kB/s | 44 kB 00:00 2023-12-08T10:02:07.887 INFO:teuthology.orchestra.run.smithi170.stdout:(8/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.6 MB/s | 80 kB 00:00 2023-12-08T10:02:07.912 INFO:teuthology.orchestra.run.smithi170.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.2 MB/s | 213 kB 00:00 2023-12-08T10:02:07.938 INFO:teuthology.orchestra.run.smithi170.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 387 kB 00:00 2023-12-08T10:02:07.963 INFO:teuthology.orchestra.run.smithi170.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.1 MB/s | 368 kB 00:00 2023-12-08T10:02:07.988 INFO:teuthology.orchestra.run.smithi170.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2023-12-08T10:02:08.064 INFO:teuthology.orchestra.run.smithi170.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.4 MB/s | 145 kB 00:00 2023-12-08T10:02:08.089 INFO:teuthology.orchestra.run.smithi170.stdout:(14/27): librdkafka-0.11.4-3.el8.x86_64.rpm 3.5 MB/s | 354 kB 00:00 2023-12-08T10:02:08.114 INFO:teuthology.orchestra.run.smithi170.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-12-08T10:02:08.139 INFO:teuthology.orchestra.run.smithi170.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 874 kB/s | 44 kB 00:00 2023-12-08T10:02:08.357 INFO:teuthology.orchestra.run.smithi170.stdout:(17/27): ceph-common-16.2.4-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2023-12-08T10:02:08.607 INFO:teuthology.orchestra.run.smithi170.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 86 kB/s | 42 kB 00:00 2023-12-08T10:02:08.624 INFO:teuthology.orchestra.run.smithi170.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 97 kB/s | 47 kB 00:00 2023-12-08T10:02:08.690 INFO:teuthology.orchestra.run.smithi170.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 240 kB/s | 80 kB 00:00 2023-12-08T10:02:08.749 INFO:teuthology.orchestra.run.smithi170.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 930 kB/s | 116 kB 00:00 2023-12-08T10:02:08.783 INFO:teuthology.orchestra.run.smithi170.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2023-12-08T10:02:08.808 INFO:teuthology.orchestra.run.smithi170.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-12-08T10:02:08.834 INFO:teuthology.orchestra.run.smithi170.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-12-08T10:02:08.859 INFO:teuthology.orchestra.run.smithi170.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2023-12-08T10:02:08.865 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:02:08.868 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-08T10:02:08.869 INFO:teuthology.orchestra.run.smithi023.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-08T10:02:08.870 INFO:teuthology.orchestra.run.smithi023.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout:Install 25 Packages 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 2 Packages 2023-12-08T10:02:08.871 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:08.872 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 59 M 2023-12-08T10:02:08.872 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:02:08.951 INFO:teuthology.orchestra.run.smithi170.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.0 MB 00:00 2023-12-08T10:02:09.010 INFO:teuthology.orchestra.run.smithi170.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-12-08T10:02:09.012 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:02:09.012 INFO:teuthology.orchestra.run.smithi170.stdout:Total 23 MB/s | 59 MB 00:02 2023-12-08T10:02:09.078 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:02:09.126 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:02:09.126 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:02:09.423 INFO:teuthology.orchestra.run.smithi023.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 40 MB/s | 6.0 MB 00:00 2023-12-08T10:02:09.448 INFO:teuthology.orchestra.run.smithi023.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2023-12-08T10:02:09.473 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:02:09.476 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:02:09.482 INFO:teuthology.orchestra.run.smithi023.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-12-08T10:02:09.515 INFO:teuthology.orchestra.run.smithi023.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-08T10:02:09.649 INFO:teuthology.orchestra.run.smithi023.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 31 MB/s | 4.1 MB 00:00 2023-12-08T10:02:09.674 INFO:teuthology.orchestra.run.smithi023.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-08T10:02:09.708 INFO:teuthology.orchestra.run.smithi023.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.3 MB/s | 80 kB 00:00 2023-12-08T10:02:09.724 INFO:teuthology.orchestra.run.smithi023.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 13 MB/s | 213 kB 00:00 2023-12-08T10:02:09.758 INFO:teuthology.orchestra.run.smithi023.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-08T10:02:09.792 INFO:teuthology.orchestra.run.smithi023.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-12-08T10:02:09.817 INFO:teuthology.orchestra.run.smithi023.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-08T10:02:09.950 INFO:teuthology.orchestra.run.smithi023.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2023-12-08T10:02:10.218 INFO:teuthology.orchestra.run.smithi023.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2023-12-08T10:02:10.266 INFO:teuthology.orchestra.run.smithi023.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 323 kB/s | 145 kB 00:00 2023-12-08T10:02:10.293 INFO:teuthology.orchestra.run.smithi023.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-12-08T10:02:10.318 INFO:teuthology.orchestra.run.smithi023.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 846 kB/s | 44 kB 00:00 2023-12-08T10:02:10.377 INFO:teuthology.orchestra.run.smithi023.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2023-12-08T10:02:10.572 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:02:10.661 INFO:teuthology.orchestra.run.smithi023.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 115 kB/s | 42 kB 00:00 2023-12-08T10:02:10.686 INFO:teuthology.orchestra.run.smithi023.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 128 kB/s | 47 kB 00:00 2023-12-08T10:02:10.711 INFO:teuthology.orchestra.run.smithi023.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 240 kB/s | 80 kB 00:00 2023-12-08T10:02:10.798 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-08T10:02:10.929 INFO:teuthology.orchestra.run.smithi023.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 481 kB/s | 116 kB 00:00 2023-12-08T10:02:10.953 INFO:teuthology.orchestra.run.smithi023.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 748 kB/s | 181 kB 00:00 2023-12-08T10:02:10.979 INFO:teuthology.orchestra.run.smithi023.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-12-08T10:02:10.983 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-08T10:02:11.004 INFO:teuthology.orchestra.run.smithi023.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 782 kB/s | 59 kB 00:00 2023-12-08T10:02:11.029 INFO:teuthology.orchestra.run.smithi023.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 994 kB/s | 75 kB 00:00 2023-12-08T10:02:11.121 INFO:teuthology.orchestra.run.smithi023.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.0 MB 00:00 2023-12-08T10:02:11.146 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-08T10:02:11.180 INFO:teuthology.orchestra.run.smithi023.stdout:(27/27): librados2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-12-08T10:02:11.181 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:02:11.182 INFO:teuthology.orchestra.run.smithi023.stdout:Total 25 MB/s | 59 MB 00:02 2023-12-08T10:02:11.253 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:02:11.305 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:02:11.305 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:02:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-08T10:02:11.666 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:02:11.669 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:02:11.738 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-08T10:02:11.834 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-08T10:02:12.056 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-08T10:02:12.229 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-08T10:02:12.297 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-08T10:02:12.511 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-08T10:02:12.687 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-08T10:02:12.824 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:02:13.088 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-08T10:02:13.310 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-08T10:02:13.319 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-08T10:02:13.479 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-08T10:02:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-08T10:02:13.554 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-08T10:02:13.717 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-08T10:02:13.912 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-08T10:02:13.925 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-08T10:02:13.961 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-08T10:02:14.126 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-08T10:02:14.142 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-08T10:02:14.241 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-08T10:02:14.288 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-08T10:02:14.470 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-08T10:02:14.493 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-08T10:02:14.546 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-08T10:02:14.659 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-08T10:02:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-08T10:02:14.726 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-08T10:02:14.943 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-08T10:02:15.100 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-08T10:02:15.146 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-08T10:02:15.328 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-08T10:02:15.435 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-08T10:02:15.532 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-08T10:02:15.607 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-08T10:02:15.692 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-08T10:02:15.767 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-08T10:02:15.814 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-08T10:02:15.939 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-08T10:02:15.953 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-08T10:02:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-08T10:02:16.166 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-08T10:02:16.221 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-08T10:02:16.224 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-08T10:02:16.390 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-08T10:02:16.395 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-08T10:02:16.577 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-08T10:02:16.577 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-08T10:02:16.784 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-08T10:02:16.812 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-08T10:02:16.860 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-08T10:02:17.000 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-08T10:02:17.464 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-08T10:02:17.658 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-08T10:02:17.750 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-08T10:02:17.955 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-08T10:02:18.171 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-08T10:02:18.308 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-08T10:02:18.376 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-08T10:02:18.570 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-08T10:02:18.750 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-08T10:02:18.927 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-08T10:02:19.134 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-08T10:02:19.342 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-08T10:02:19.570 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-08T10:02:20.270 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-08T10:02:20.574 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-08T10:02:20.687 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-08T10:02:20.836 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-08T10:02:21.840 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-08T10:02:22.067 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-08T10:02:22.810 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-08T10:02:23.105 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-08T10:02:23.215 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-08T10:02:23.357 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /dev 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /run 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2023-12-08T10:02:38.967 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /dev 2023-12-08T10:02:38.968 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:40.297 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-08T10:02:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-08T10:02:40.370 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-08T10:02:40.569 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-08T10:02:40.569 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-08T10:02:41.245 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-08T10:02:41.246 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-08T10:02:41.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-08T10:02:41.247 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-08T10:02:42.284 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /run 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2023-12-08T10:02:42.285 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:42.380 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-08T10:02:42.381 INFO:teuthology.orchestra.run.smithi170.stdout: leveldb-1.22-1.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.382 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-08T10:02:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:42.383 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:42.383 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:02:42.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2023-12-08T10:02:43.466 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:42 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:02:43.710 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-08T10:02:43.791 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-08T10:02:43.792 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-08T10:02:44.007 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-08T10:02:44.007 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-08T10:02:44.733 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-08T10:02:44.734 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-08T10:02:44.735 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-08T10:02:45.152 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:02:45.153 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:45.153 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:45.154 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-08T10:02:45.155 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:45.155 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 53 M 2023-12-08T10:02:45.155 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 229 M 2023-12-08T10:02:45.155 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:02:45.287 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 169 kB 00:00 2023-12-08T10:02:45.379 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.2 MB/s | 203 kB 00:00 2023-12-08T10:02:45.421 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-12-08T10:02:45.455 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-12-08T10:02:45.488 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2023-12-08T10:02:45.903 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-08T10:02:45.903 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:45.903 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: leveldb-1.22-1.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-08T10:02:45.904 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.905 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-08T10:02:45.906 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:45.906 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:02:46.073 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2023-12-08T10:02:46.074 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:02:46.074 INFO:teuthology.orchestra.run.smithi170.stdout:Total 57 MB/s | 53 MB 00:00 2023-12-08T10:02:46.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-test 2023-12-08T10:02:46.117 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:02:46.139 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:02:46.139 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:02:46.454 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:02:46.457 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:02:46.926 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:02:46.976 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:43 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:02:47.151 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-08T10:02:47.343 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-08T10:02:47.482 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-08T10:02:47.543 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-08T10:02:47.730 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-08T10:02:47.898 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-08T10:02:48.800 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:02:48.801 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:02:48.801 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout:Install 6 Packages 2023-12-08T10:02:48.802 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:48.803 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 53 M 2023-12-08T10:02:48.803 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 229 M 2023-12-08T10:02:48.803 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:02:48.907 INFO:teuthology.orchestra.run.smithi023.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 169 kB 00:00 2023-12-08T10:02:48.940 INFO:teuthology.orchestra.run.smithi023.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.4 MB/s | 203 kB 00:00 2023-12-08T10:02:49.007 INFO:teuthology.orchestra.run.smithi023.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2023-12-08T10:02:49.032 INFO:teuthology.orchestra.run.smithi023.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-12-08T10:02:49.057 INFO:teuthology.orchestra.run.smithi023.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2023-12-08T10:02:49.633 INFO:teuthology.orchestra.run.smithi023.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2023-12-08T10:02:49.635 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:02:49.635 INFO:teuthology.orchestra.run.smithi023.stdout:Total 63 MB/s | 53 MB 00:00 2023-12-08T10:02:49.680 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:02:49.704 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:02:49.704 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:02:50.026 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:02:50.029 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:02:50.522 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:02:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-08T10:02:50.949 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-08T10:02:51.129 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-08T10:02:51.197 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-08T10:02:51.395 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-08T10:02:51.529 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-08T10:02:52.941 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-08T10:02:53.171 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-08T10:02:53.443 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-08T10:02:53.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-08T10:02:53.443 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-08T10:02:53.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-08T10:02:53.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-08T10:02:53.444 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-08T10:02:54.160 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:54.161 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:02:54.377 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2023-12-08T10:02:55.237 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:54 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:02:56.804 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-08T10:02:56.921 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:02:56.925 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-08T10:02:56.926 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-08T10:02:56.927 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout:Install 32 Packages 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:02:56.928 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 M 2023-12-08T10:02:56.929 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 111 M 2023-12-08T10:02:56.929 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:02:57.052 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-08T10:02:57.361 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-08T10:02:57.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-08T10:02:57.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-08T10:02:57.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-08T10:02:57.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-08T10:02:57.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-08T10:02:57.549 INFO:teuthology.orchestra.run.smithi170.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 123 kB/s | 6.3 kB 00:00 2023-12-08T10:02:57.600 INFO:teuthology.orchestra.run.smithi170.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-12-08T10:02:57.666 INFO:teuthology.orchestra.run.smithi170.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-12-08T10:02:57.700 INFO:teuthology.orchestra.run.smithi170.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-12-08T10:02:57.792 INFO:teuthology.orchestra.run.smithi170.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2023-12-08T10:02:57.817 INFO:teuthology.orchestra.run.smithi170.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2023-12-08T10:02:57.851 INFO:teuthology.orchestra.run.smithi170.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2023-12-08T10:02:57.876 INFO:teuthology.orchestra.run.smithi170.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 464 kB/s | 39 kB 00:00 2023-12-08T10:02:57.902 INFO:teuthology.orchestra.run.smithi170.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2023-12-08T10:02:57.960 INFO:teuthology.orchestra.run.smithi170.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.3 MB/s | 457 kB 00:00 2023-12-08T10:02:58.144 INFO:teuthology.orchestra.run.smithi170.stdout:(11/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 33 MB/s | 18 MB 00:00 2023-12-08T10:02:58.174 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-08T10:02:58.174 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:58.174 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:02:58.174 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-08T10:02:58.174 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-08T10:02:58.174 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-08T10:02:58.175 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:02:58.175 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:02:58.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph 2023-12-08T10:02:58.570 INFO:teuthology.orchestra.run.smithi170.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 373 kB/s | 249 kB 00:00 2023-12-08T10:02:58.595 INFO:teuthology.orchestra.run.smithi170.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 242 kB/s | 109 kB 00:00 2023-12-08T10:02:58.795 INFO:teuthology.orchestra.run.smithi170.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 768 kB/s | 641 kB 00:00 2023-12-08T10:02:58.903 INFO:teuthology.orchestra.run.smithi170.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 131 kB/s | 44 kB 00:00 2023-12-08T10:02:59.054 INFO:teuthology.orchestra.run.smithi170.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 404 kB/s | 185 kB 00:00 2023-12-08T10:02:59.154 INFO:teuthology.orchestra.run.smithi170.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-12-08T10:02:59.229 INFO:teuthology.orchestra.run.smithi170.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.2 MB/s | 384 kB 00:00 2023-12-08T10:02:59.255 INFO:teuthology.orchestra.run.smithi170.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 376 kB/s | 173 kB 00:00 2023-12-08T10:02:59.280 INFO:teuthology.orchestra.run.smithi170.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2023-12-08T10:02:59.304 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:56 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:02:59.330 INFO:teuthology.orchestra.run.smithi170.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 484 kB/s | 49 kB 00:00 2023-12-08T10:02:59.364 INFO:teuthology.orchestra.run.smithi170.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 538 kB/s | 59 kB 00:00 2023-12-08T10:02:59.431 INFO:teuthology.orchestra.run.smithi170.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-12-08T10:02:59.464 INFO:teuthology.orchestra.run.smithi170.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2023-12-08T10:02:59.490 INFO:teuthology.orchestra.run.smithi170.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-12-08T10:02:59.535 INFO:teuthology.orchestra.run.smithi170.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-12-08T10:02:59.565 INFO:teuthology.orchestra.run.smithi170.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-12-08T10:02:59.591 INFO:teuthology.orchestra.run.smithi170.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2023-12-08T10:02:59.657 INFO:teuthology.orchestra.run.smithi170.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-12-08T10:02:59.699 INFO:teuthology.orchestra.run.smithi170.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 787 kB/s | 85 kB 00:00 2023-12-08T10:02:59.733 INFO:teuthology.orchestra.run.smithi170.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-12-08T10:02:59.758 INFO:teuthology.orchestra.run.smithi170.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2023-12-08T10:02:59.759 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:02:59.760 INFO:teuthology.orchestra.run.smithi170.stdout:Total 11 MB/s | 31 MB 00:02 2023-12-08T10:02:59.844 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:02:59.882 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:02:59.883 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:03:00.203 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:03:00.206 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:03:01.095 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:01.098 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:01.098 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-08T10:03:01.099 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-08T10:03:01.100 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout:Install 32 Packages 2023-12-08T10:03:01.101 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:01.102 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 31 M 2023-12-08T10:03:01.102 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 111 M 2023-12-08T10:03:01.102 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:03:01.287 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:03:01.492 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-08T10:03:01.721 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-08T10:03:01.726 INFO:teuthology.orchestra.run.smithi023.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 125 kB/s | 6.3 kB 00:00 2023-12-08T10:03:01.777 INFO:teuthology.orchestra.run.smithi023.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2023-12-08T10:03:01.827 INFO:teuthology.orchestra.run.smithi023.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2023-12-08T10:03:01.853 INFO:teuthology.orchestra.run.smithi023.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-12-08T10:03:01.936 INFO:teuthology.orchestra.run.smithi023.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2023-12-08T10:03:01.970 INFO:teuthology.orchestra.run.smithi023.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2023-12-08T10:03:01.979 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-08T10:03:02.003 INFO:teuthology.orchestra.run.smithi023.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2023-12-08T10:03:02.029 INFO:teuthology.orchestra.run.smithi023.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 421 kB/s | 39 kB 00:00 2023-12-08T10:03:02.054 INFO:teuthology.orchestra.run.smithi023.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2023-12-08T10:03:02.138 INFO:teuthology.orchestra.run.smithi023.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.1 MB/s | 457 kB 00:00 2023-12-08T10:03:02.203 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-08T10:03:02.313 INFO:teuthology.orchestra.run.smithi023.stdout:(11/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 34 MB/s | 18 MB 00:00 2023-12-08T10:03:02.358 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-08T10:03:02.531 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-08T10:03:02.580 INFO:teuthology.orchestra.run.smithi023.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 473 kB/s | 249 kB 00:00 2023-12-08T10:03:02.682 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-08T10:03:02.780 INFO:teuthology.orchestra.run.smithi023.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 234 kB/s | 109 kB 00:00 2023-12-08T10:03:02.852 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-08T10:03:02.964 INFO:teuthology.orchestra.run.smithi023.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 114 kB/s | 44 kB 00:00 2023-12-08T10:03:03.044 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-08T10:03:03.072 INFO:teuthology.orchestra.run.smithi023.stdout:(15/32): python3-cryptography-3.2.1-6.el8.x86_6 686 kB/s | 641 kB 00:00 2023-12-08T10:03:03.239 INFO:teuthology.orchestra.run.smithi023.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 629 kB/s | 173 kB 00:00 2023-12-08T10:03:03.265 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-08T10:03:03.323 INFO:teuthology.orchestra.run.smithi023.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 341 kB/s | 185 kB 00:00 2023-12-08T10:03:03.348 INFO:teuthology.orchestra.run.smithi023.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 103 kB/s | 11 kB 00:00 2023-12-08T10:03:03.434 INFO:teuthology.orchestra.run.smithi023.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2023-12-08T10:03:03.457 INFO:teuthology.orchestra.run.smithi023.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 449 kB/s | 49 kB 00:00 2023-12-08T10:03:03.486 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-08T10:03:03.549 INFO:teuthology.orchestra.run.smithi023.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 502 kB/s | 59 kB 00:00 2023-12-08T10:03:03.649 INFO:teuthology.orchestra.run.smithi023.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-12-08T10:03:03.674 INFO:teuthology.orchestra.run.smithi023.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-12-08T10:03:03.708 INFO:teuthology.orchestra.run.smithi023.stdout:(24/32): python3-cherrypy-18.4.0-1.el8.noarch.r 605 kB/s | 384 kB 00:00 2023-12-08T10:03:03.728 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-08T10:03:03.742 INFO:teuthology.orchestra.run.smithi023.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2023-12-08T10:03:03.767 INFO:teuthology.orchestra.run.smithi023.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 260 kB/s | 24 kB 00:00 2023-12-08T10:03:03.809 INFO:teuthology.orchestra.run.smithi023.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 290 kB/s | 29 kB 00:00 2023-12-08T10:03:03.835 INFO:teuthology.orchestra.run.smithi023.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2023-12-08T10:03:03.893 INFO:teuthology.orchestra.run.smithi023.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-12-08T10:03:03.943 INFO:teuthology.orchestra.run.smithi023.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2023-12-08T10:03:03.968 INFO:teuthology.orchestra.run.smithi023.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 638 kB/s | 85 kB 00:00 2023-12-08T10:03:03.985 INFO:teuthology.orchestra.run.smithi023.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 250 kB/s | 23 kB 00:00 2023-12-08T10:03:03.989 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:03.989 INFO:teuthology.orchestra.run.smithi023.stdout:Total 11 MB/s | 31 MB 00:02 2023-12-08T10:03:04.076 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:03:04.107 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-08T10:03:04.117 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:03:04.117 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:03:04.373 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-08T10:03:04.440 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:03:04.444 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:03:04.537 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-08T10:03:04.803 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-08T10:03:04.988 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-08T10:03:05.291 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-08T10:03:05.493 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-08T10:03:05.628 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:03:05.682 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-08T10:03:05.828 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-08T10:03:05.888 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-08T10:03:06.043 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-08T10:03:06.132 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-08T10:03:06.272 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-08T10:03:06.364 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-08T10:03:06.466 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-08T10:03:06.584 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-08T10:03:06.648 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-08T10:03:06.812 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-08T10:03:06.895 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-08T10:03:06.962 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-08T10:03:07.132 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-08T10:03:07.191 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-08T10:03:07.350 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-08T10:03:07.545 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-08T10:03:07.604 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-08T10:03:07.742 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-08T10:03:07.825 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-08T10:03:07.991 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-08T10:03:08.350 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-08T10:03:08.622 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-08T10:03:08.776 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-08T10:03:08.997 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-08T10:03:09.194 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-08T10:03:09.350 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-08T10:03:09.474 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-08T10:03:09.558 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-08T10:03:09.682 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-08T10:03:09.904 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-08T10:03:09.989 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-08T10:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-08T10:03:10.199 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-08T10:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-08T10:03:10.421 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-08T10:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-08T10:03:10.587 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-08T10:03:10.604 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-08T10:03:10.728 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-08T10:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-08T10:03:11.124 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-08T10:03:11.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-08T10:03:11.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-08T10:03:11.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-08T10:03:11.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-08T10:03:11.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-08T10:03:11.125 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-08T10:03:11.126 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-08T10:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-08T10:03:11.127 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-08T10:03:11.127 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-08T10:03:11.127 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-08T10:03:11.414 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-08T10:03:11.854 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-08T10:03:12.073 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-08T10:03:12.194 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-08T10:03:12.194 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:12.194 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:03:12.194 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-08T10:03:12.195 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-2.6-1.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-08T10:03:12.196 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-08T10:03:12.197 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-08T10:03:12.197 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-08T10:03:12.197 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-08T10:03:12.197 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:12.197 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:12.342 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2023-12-08T10:03:13.203 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:12 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:13.708 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-08T10:03:13.881 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-08T10:03:14.342 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-08T10:03:14.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-08T10:03:14.657 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:03:14.800 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-08T10:03:14.880 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:14.881 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:03:14.881 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:14.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2023-12-08T10:03:15.001 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-08T10:03:15.019 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-08T10:03:15.167 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-08T10:03:15.599 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-08T10:03:15.600 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-08T10:03:15.601 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-08T10:03:15.602 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-08T10:03:15.602 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-08T10:03:15.602 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-08T10:03:15.602 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-08T10:03:15.602 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-08T10:03:15.602 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-08T10:03:15.838 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:14 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:16.726 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-08T10:03:16.727 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-2.6-1.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-08T10:03:16.728 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-08T10:03:16.729 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-08T10:03:16.729 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-08T10:03:16.729 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-08T10:03:16.729 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:16.729 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:16.865 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-base 2023-12-08T10:03:17.516 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 71 k 2023-12-08T10:03:17.517 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 290 k 2023-12-08T10:03:17.518 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:03:17.556 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2023-12-08T10:03:17.556 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:17.556 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.8 MB/s | 71 kB 00:00 2023-12-08T10:03:17.557 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:03:17.563 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:03:17.563 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:03:17.565 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:03:17.568 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:03:17.610 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:03:17.765 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:14 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:18.101 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:18.302 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:18.528 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:19.193 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:19.193 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:19.193 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:03:19.193 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-08T10:03:19.193 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:19.193 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:19.316 INFO:teuthology.orchestra.run.smithi023.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:03:19.557 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:19.558 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:03:19.558 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:19.663 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install cephadm 2023-12-08T10:03:20.556 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:17 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:20.738 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2023-12-08T10:03:21.610 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:20 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:22.333 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:03:22.334 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:22.335 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 71 k 2023-12-08T10:03:22.335 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 290 k 2023-12-08T10:03:22.335 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:03:22.381 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2023-12-08T10:03:22.382 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:22.382 INFO:teuthology.orchestra.run.smithi023.stdout:Total 1.5 MB/s | 71 kB 00:00 2023-12-08T10:03:22.382 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:03:22.390 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:03:22.390 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:03:22.392 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:03:22.395 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:03:22.438 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:03:23.273 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:23.273 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:23.273 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:03:23.274 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:23.275 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 163 k 2023-12-08T10:03:23.275 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 587 k 2023-12-08T10:03:23.275 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:03:23.320 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.5 MB/s | 163 kB 00:00 2023-12-08T10:03:23.321 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:23.321 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.4 MB/s | 163 kB 00:00 2023-12-08T10:03:23.321 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:03:23.332 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:03:23.332 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:03:23.380 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:03:23.383 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:03:23.499 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:03:23.684 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:03:23.754 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:23.984 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:24.104 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:03:24.229 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:03:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:24.772 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:03:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:24.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:24.772 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:24.925 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-08T10:03:24.926 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:24.926 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:03:24.926 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-08T10:03:24.926 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:24.926 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:24.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2023-12-08T10:03:25.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-immutable-object-cache 2023-12-08T10:03:25.847 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:24 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:26.028 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:23 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:27.307 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:03:27.530 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:27.531 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:03:27.531 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:27.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2023-12-08T10:03:27.829 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repo Size 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:27.830 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:03:27.831 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:27.831 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 163 k 2023-12-08T10:03:27.831 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 587 k 2023-12-08T10:03:27.831 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:03:27.870 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.2 MB/s | 163 kB 00:00 2023-12-08T10:03:27.870 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:27.871 INFO:teuthology.orchestra.run.smithi023.stdout:Total 4.0 MB/s | 163 kB 00:00 2023-12-08T10:03:27.871 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:03:27.882 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:03:27.882 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:03:27.932 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:03:27.935 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:03:28.047 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:03:28.215 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:03:28.507 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:27 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:28.646 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:03:29.979 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:03:29.979 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:29.979 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:03:29.979 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-08T10:03:29.979 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:29.979 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:30.206 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:30.208 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:03:30.209 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:30.209 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-08T10:03:30.209 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:30.209 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.0 M 2023-12-08T10:03:30.209 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 15 M 2023-12-08T10:03:30.210 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:03:30.474 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 495 kB/s | 24 kB 00:00 2023-12-08T10:03:30.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr 2023-12-08T10:03:30.490 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 136 kB/s | 8.9 kB 00:00 2023-12-08T10:03:30.558 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2023-12-08T10:03:30.625 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 251 kB/s | 34 kB 00:00 2023-12-08T10:03:30.700 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2023-12-08T10:03:30.817 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 124 kB/s | 43 kB 00:00 2023-12-08T10:03:30.819 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:30.819 INFO:teuthology.orchestra.run.smithi170.stdout:Total 4.8 MB/s | 3.0 MB 00:00 2023-12-08T10:03:30.829 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:03:30.839 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:03:30.839 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:03:31.014 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:03:31.017 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:03:31.300 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:03:31.384 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:28 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:31.497 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-08T10:03:31.792 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-08T10:03:31.985 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-08T10:03:32.106 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-08T10:03:32.951 INFO:teuthology.orchestra.run.smithi023.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:03:33.084 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-08T10:03:33.191 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:33.192 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:03:33.192 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:33.296 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-dashboard 2023-12-08T10:03:33.353 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:03:33.642 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:03:33.643 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-08T10:03:33.643 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-08T10:03:33.643 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-08T10:03:33.643 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-08T10:03:33.643 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-08T10:03:34.195 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:31 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:34.320 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-08T10:03:34.321 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-08T10:03:34.322 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:34.322 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:34.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-08T10:03:35.283 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:34 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:36.038 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-08T10:03:36.039 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Install 6 Packages 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 3.0 M 2023-12-08T10:03:36.040 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 15 M 2023-12-08T10:03:36.041 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:03:36.631 INFO:teuthology.orchestra.run.smithi023.stdout:(1/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 136 kB/s | 8.9 kB 00:00 2023-12-08T10:03:36.657 INFO:teuthology.orchestra.run.smithi023.stdout:(2/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 263 kB/s | 24 kB 00:00 2023-12-08T10:03:36.715 INFO:teuthology.orchestra.run.smithi023.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-12-08T10:03:36.782 INFO:teuthology.orchestra.run.smithi023.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 268 kB/s | 34 kB 00:00 2023-12-08T10:03:36.832 INFO:teuthology.orchestra.run.smithi023.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2023-12-08T10:03:36.956 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================================ 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================================ 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-08T10:03:36.958 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================================ 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout:Install 8 Packages 2023-12-08T10:03:36.959 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:36.960 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2023-12-08T10:03:36.960 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 210 M 2023-12-08T10:03:36.960 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:03:37.008 INFO:teuthology.orchestra.run.smithi023.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 113 kB/s | 43 kB 00:00 2023-12-08T10:03:37.009 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:37.010 INFO:teuthology.orchestra.run.smithi023.stdout:Total 3.0 MB/s | 3.0 MB 00:00 2023-12-08T10:03:37.021 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:03:37.032 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:03:37.032 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:03:37.207 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:03:37.211 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:03:37.265 INFO:teuthology.orchestra.run.smithi170.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 38 MB/s | 8.6 MB 00:00 2023-12-08T10:03:37.478 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:03:37.524 INFO:teuthology.orchestra.run.smithi170.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 9.8 MB/s | 4.7 MB 00:00 2023-12-08T10:03:37.591 INFO:teuthology.orchestra.run.smithi170.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.8 MB/s | 4.9 MB 00:00 2023-12-08T10:03:37.617 INFO:teuthology.orchestra.run.smithi170.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-12-08T10:03:37.706 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-08T10:03:37.717 INFO:teuthology.orchestra.run.smithi170.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 8.3 MB/s | 3.7 MB 00:00 2023-12-08T10:03:37.933 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-08T10:03:38.075 INFO:teuthology.orchestra.run.smithi170.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2023-12-08T10:03:38.124 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-08T10:03:38.125 INFO:teuthology.orchestra.run.smithi170.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 422 kB/s | 172 kB 00:00 2023-12-08T10:03:38.268 INFO:teuthology.orchestra.run.smithi170.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 992 kB/s | 645 kB 00:00 2023-12-08T10:03:38.268 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:38.268 INFO:teuthology.orchestra.run.smithi170.stdout:Total 28 MB/s | 37 MB 00:01 2023-12-08T10:03:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-08T10:03:38.497 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:03:38.516 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:03:38.516 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:03:38.838 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:03:38.841 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:03:39.321 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-08T10:03:39.432 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:03:39.565 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-08T10:03:39.610 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:03:39.685 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-08T10:03:39.840 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-08T10:03:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:03:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-08T10:03:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-08T10:03:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-08T10:03:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-08T10:03:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-08T10:03:40.440 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-08T10:03:40.623 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-08T10:03:40.633 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-08T10:03:40.633 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:40.633 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:03:40.633 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:40.634 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:41.010 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-08T10:03:41.234 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-08T10:03:41.398 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-08T10:03:41.894 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:38 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:42.095 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-08T10:03:42.976 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-08T10:03:43.433 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-08T10:03:43.678 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================================ 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================================ 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-08T10:03:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================================ 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout:Install 8 Packages 2023-12-08T10:03:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:43.681 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 37 M 2023-12-08T10:03:43.681 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 210 M 2023-12-08T10:03:43.681 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:03:44.097 INFO:teuthology.orchestra.run.smithi023.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 51 MB/s | 8.6 MB 00:00 2023-12-08T10:03:44.447 INFO:teuthology.orchestra.run.smithi023.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.4 MB/s | 4.9 MB 00:00 2023-12-08T10:03:44.515 INFO:teuthology.orchestra.run.smithi023.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 8.1 MB/s | 4.7 MB 00:00 2023-12-08T10:03:44.540 INFO:teuthology.orchestra.run.smithi023.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-12-08T10:03:44.882 INFO:teuthology.orchestra.run.smithi023.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.8 MB/s | 3.7 MB 00:00 2023-12-08T10:03:45.024 INFO:teuthology.orchestra.run.smithi023.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 27 MB/s | 14 MB 00:00 2023-12-08T10:03:45.186 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-08T10:03:45.224 INFO:teuthology.orchestra.run.smithi023.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 943 kB/s | 645 kB 00:00 2023-12-08T10:03:45.358 INFO:teuthology.orchestra.run.smithi023.stdout:(8/8): libquadmath-8.5.0-21.el8.x86_64.rpm 363 kB/s | 172 kB 00:00 2023-12-08T10:03:45.358 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:45.358 INFO:teuthology.orchestra.run.smithi023.stdout:Total 22 MB/s | 37 MB 00:01 2023-12-08T10:03:45.597 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:03:45.616 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:03:45.616 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:03:45.967 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:03:45.970 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:03:46.481 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-08T10:03:46.724 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-08T10:03:46.780 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:03:47.039 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-08T10:03:47.039 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-08T10:03:47.039 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-08T10:03:47.039 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-08T10:03:47.040 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-08T10:03:47.040 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-08T10:03:47.040 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-08T10:03:47.040 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-08T10:03:47.412 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-08T10:03:47.539 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-08T10:03:47.695 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-08T10:03:47.738 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:47.739 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:47.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2023-12-08T10:03:48.327 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-08T10:03:48.504 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-08T10:03:48.711 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:47 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:49.165 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-08T10:03:49.345 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-08T10:03:50.077 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-08T10:03:50.392 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-08T10:03:50.394 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:03:50.395 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2023-12-08T10:03:50.396 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:50.396 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 1.9 M 2023-12-08T10:03:50.396 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-12-08T10:03:50.396 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:03:50.411 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-08T10:03:50.695 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-08T10:03:50.893 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 558 kB/s | 27 kB 00:00 2023-12-08T10:03:50.943 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 270 kB/s | 27 kB 00:00 2023-12-08T10:03:50.969 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 161 kB/s | 20 kB 00:00 2023-12-08T10:03:50.994 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2023-12-08T10:03:51.019 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2023-12-08T10:03:51.278 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 128 kB/s | 33 kB 00:00 2023-12-08T10:03:51.328 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 128 kB/s | 43 kB 00:00 2023-12-08T10:03:51.370 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 177 kB/s | 16 kB 00:00 2023-12-08T10:03:51.412 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 350 kB/s | 155 kB 00:00 2023-12-08T10:03:51.638 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 263 kB/s | 81 kB 00:00 2023-12-08T10:03:51.713 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 214 kB/s | 64 kB 00:00 2023-12-08T10:03:51.746 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 3.1 MB/s | 1.2 MB 00:00 2023-12-08T10:03:51.771 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 459 kB/s | 61 kB 00:00 2023-12-08T10:03:51.774 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:51.774 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.4 MB/s | 1.9 MB 00:01 2023-12-08T10:03:51.834 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:03:51.849 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:03:51.850 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:03:51.938 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:03:51.941 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:03:52.434 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:03:52.575 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-08T10:03:52.688 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-08T10:03:52.896 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-08T10:03:53.140 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-08T10:03:53.386 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-08T10:03:53.632 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-08T10:03:53.851 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-08T10:03:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-08T10:03:54.104 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-08T10:03:54.153 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-08T10:03:54.401 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-08T10:03:54.446 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-08T10:03:55.208 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-08T10:03:55.208 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:55.208 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:03:55.208 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-08T10:03:55.208 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:55.209 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:03:55.330 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-rook 2023-12-08T10:03:55.469 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-08T10:03:55.744 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-08T10:03:55.953 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-08T10:03:56.153 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-08T10:03:56.223 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:01:53 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:03:56.337 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-08T10:03:56.626 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-08T10:03:56.627 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-08T10:03:56.627 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-08T10:03:56.627 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-08T10:03:56.627 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-08T10:03:56.627 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-08T10:03:56.627 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-08T10:03:57.378 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:03:57.379 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:03:57.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2023-12-08T10:03:58.033 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:03:58.035 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:58.035 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:03:58.035 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:58.035 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-08T10:03:58.036 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout:Install 13 Packages 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:03:58.037 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 1.9 M 2023-12-08T10:03:58.038 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 24 M 2023-12-08T10:03:58.038 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:03:58.348 INFO:teuthology.orchestra.run.smithi023.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 604 kB/s | 27 kB 00:00 2023-12-08T10:03:58.361 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:57 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:03:58.390 INFO:teuthology.orchestra.run.smithi023.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 231 kB/s | 20 kB 00:00 2023-12-08T10:03:58.416 INFO:teuthology.orchestra.run.smithi023.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 238 kB/s | 27 kB 00:00 2023-12-08T10:03:58.441 INFO:teuthology.orchestra.run.smithi023.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2023-12-08T10:03:58.458 INFO:teuthology.orchestra.run.smithi023.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.6 MB/s | 110 kB 00:00 2023-12-08T10:03:58.766 INFO:teuthology.orchestra.run.smithi023.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2023-12-08T10:03:58.825 INFO:teuthology.orchestra.run.smithi023.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 90 kB/s | 33 kB 00:00 2023-12-08T10:03:58.850 INFO:teuthology.orchestra.run.smithi023.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 356 kB/s | 155 kB 00:00 2023-12-08T10:03:59.059 INFO:teuthology.orchestra.run.smithi023.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 348 kB/s | 81 kB 00:00 2023-12-08T10:03:59.109 INFO:teuthology.orchestra.run.smithi023.stdout:(10/13): python3-certifi-2018.10.15-7.el8.noarc 48 kB/s | 16 kB 00:00 2023-12-08T10:03:59.201 INFO:teuthology.orchestra.run.smithi023.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 452 kB/s | 64 kB 00:00 2023-12-08T10:03:59.326 INFO:teuthology.orchestra.run.smithi023.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 282 kB/s | 61 kB 00:00 2023-12-08T10:03:59.618 INFO:teuthology.orchestra.run.smithi023.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.5 MB/s | 1.2 MB 00:00 2023-12-08T10:03:59.621 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:03:59.621 INFO:teuthology.orchestra.run.smithi023.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2023-12-08T10:03:59.683 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:03:59.701 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:03:59.701 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:03:59.794 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:03:59.798 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:00.034 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:00.035 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:00.035 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2023-12-08T10:04:00.035 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2023-12-08T10:04:00.036 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:00.037 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 5.8 M 2023-12-08T10:04:00.037 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 24 M 2023-12-08T10:04:00.037 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:00.295 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2023-12-08T10:04:00.330 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:00.421 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-12-08T10:04:00.563 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-08T10:04:00.638 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 62 kB/s | 21 kB 00:00 2023-12-08T10:04:00.738 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.8 MB/s | 4.8 MB 00:00 2023-12-08T10:04:00.778 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-08T10:04:00.822 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 266 kB/s | 49 kB 00:00 2023-12-08T10:04:00.997 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-08T10:04:01.030 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 584 kB/s | 355 kB 00:00 2023-12-08T10:04:01.032 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:01.033 INFO:teuthology.orchestra.run.smithi170.stdout:Total 5.8 MB/s | 5.8 MB 00:00 2023-12-08T10:04:01.084 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:01.093 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:01.093 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:01.195 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:01.197 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:01.217 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-08T10:04:01.463 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-08T10:04:01.471 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:01.692 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-08T10:04:01.727 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-08T10:04:01.970 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-08T10:04:02.052 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-08T10:04:02.225 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-08T10:04:02.996 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-08T10:04:03.312 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-08T10:04:03.340 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-08T10:04:03.557 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-08T10:04:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-08T10:04:03.761 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:04:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-08T10:04:04.010 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-08T10:04:04.165 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:04:04.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-08T10:04:04.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-08T10:04:04.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-08T10:04:04.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-08T10:04:04.165 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-08T10:04:04.176 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-08T10:04:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-08T10:04:04.473 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-08T10:04:04.862 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:04.863 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:04.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2023-12-08T10:04:05.281 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-08T10:04:05.281 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:05.281 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-08T10:04:05.282 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-08T10:04:05.283 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-08T10:04:05.283 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:05.283 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:05.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-mgr-cephadm 2023-12-08T10:04:05.858 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:04 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:06.294 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:03 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-08T10:04:07.541 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 968 k 2023-12-08T10:04:07.542 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 3.2 M 2023-12-08T10:04:07.543 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:07.792 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-12-08T10:04:08.075 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 245 kB/s | 83 kB 00:00 2023-12-08T10:04:08.076 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:08.076 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.8 MB/s | 968 kB 00:00 2023-12-08T10:04:08.086 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:08.107 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:08.107 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:08.107 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:08.108 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2023-12-08T10:04:08.108 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:08.108 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-08T10:04:08.108 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout:Install 6 Packages 2023-12-08T10:04:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:08.110 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 5.8 M 2023-12-08T10:04:08.110 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 24 M 2023-12-08T10:04:08.110 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:08.192 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:08.195 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:08.360 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:08.610 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-08T10:04:08.686 INFO:teuthology.orchestra.run.smithi023.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2023-12-08T10:04:08.802 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-08T10:04:08.819 INFO:teuthology.orchestra.run.smithi023.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.8 MB/s | 538 kB 00:00 2023-12-08T10:04:08.945 INFO:teuthology.orchestra.run.smithi023.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 82 kB/s | 21 kB 00:00 2023-12-08T10:04:09.078 INFO:teuthology.orchestra.run.smithi023.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 367 kB/s | 49 kB 00:00 2023-12-08T10:04:09.145 INFO:teuthology.orchestra.run.smithi023.stdout:(5/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.3 MB/s | 4.8 MB 00:00 2023-12-08T10:04:09.194 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-08T10:04:09.195 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-08T10:04:09.245 INFO:teuthology.orchestra.run.smithi023.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 836 kB/s | 355 kB 00:00 2023-12-08T10:04:09.248 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:09.249 INFO:teuthology.orchestra.run.smithi023.stdout:Total 5.1 MB/s | 5.8 MB 00:01 2023-12-08T10:04:09.305 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:09.316 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:09.316 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:09.427 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:09.430 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:09.706 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-08T10:04:10.124 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-08T10:04:10.124 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:10.124 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:10.124 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-08T10:04:10.124 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:10.124 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-08T10:04:10.344 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2023-12-08T10:04:11.217 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:10 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-08T10:04:11.619 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-08T10:04:11.831 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-08T10:04:12.017 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:04:12.308 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-08T10:04:12.308 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-08T10:04:12.308 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-08T10:04:12.308 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-08T10:04:12.308 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-08T10:04:12.308 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-08T10:04:12.905 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:12.906 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:12.906 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:12.906 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 131 k 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 439 k 2023-12-08T10:04:12.907 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:12.956 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2023-12-08T10:04:12.957 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:12.957 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.7 MB/s | 131 kB 00:00 2023-12-08T10:04:12.957 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:12.960 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:12.960 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:13.010 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:13.013 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:13.053 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-08T10:04:13.053 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:13.053 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:13.053 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-08T10:04:13.053 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-08T10:04:13.054 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-08T10:04:13.054 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:13.054 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:13.118 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:13.196 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-fuse 2023-12-08T10:04:13.367 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:13.661 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:14.094 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:11 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:14.326 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:14.326 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:14.326 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:14.326 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:14.326 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:14.327 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:14.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2023-12-08T10:04:15.392 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:14 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:15.882 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:15.882 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:15.882 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:15.882 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:15.882 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:15.882 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout:Install 2 Packages 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:15.883 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 968 k 2023-12-08T10:04:15.884 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 3.2 M 2023-12-08T10:04:15.884 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:15.998 INFO:teuthology.orchestra.run.smithi023.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2023-12-08T10:04:16.273 INFO:teuthology.orchestra.run.smithi023.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 247 kB/s | 83 kB 00:00 2023-12-08T10:04:16.274 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:16.274 INFO:teuthology.orchestra.run.smithi023.stdout:Total 2.4 MB/s | 968 kB 00:00 2023-12-08T10:04:16.285 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:16.308 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:16.308 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:16.402 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:16.405 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:16.580 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:16.858 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:16.897 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-08T10:04:17.079 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:17.080 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:17.080 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:17.083 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-08T10:04:17.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2023-12-08T10:04:17.492 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-08T10:04:17.493 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-08T10:04:18.059 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:17 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:18.252 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-08T10:04:18.252 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:18.252 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:18.252 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-08T10:04:18.252 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:18.252 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:18.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install librados-devel 2023-12-08T10:04:19.370 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:16 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:19.743 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:19.744 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:04:19.745 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:19.745 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 24 k 2023-12-08T10:04:19.745 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 94 k 2023-12-08T10:04:19.745 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:19.794 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 496 kB/s | 24 kB 00:00 2023-12-08T10:04:19.795 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:19.795 INFO:teuthology.orchestra.run.smithi170.stdout:Total 483 kB/s | 24 kB 00:00 2023-12-08T10:04:19.795 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:19.798 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:19.798 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:19.801 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:19.804 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:19.852 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:20.032 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:20.303 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:20.945 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:20.946 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:20.946 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:20.946 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:20.946 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:20.946 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:21.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2023-12-08T10:04:21.177 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:04:21.178 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:21.179 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 131 k 2023-12-08T10:04:21.179 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 439 k 2023-12-08T10:04:21.179 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:21.235 INFO:teuthology.orchestra.run.smithi023.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 131 kB 00:00 2023-12-08T10:04:21.235 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:21.236 INFO:teuthology.orchestra.run.smithi023.stdout:Total 2.3 MB/s | 131 kB 00:00 2023-12-08T10:04:21.236 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:21.242 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:21.242 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:21.295 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:21.298 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:21.411 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:21.607 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:21.917 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:20 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:21.918 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:22.614 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:22.614 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:22.614 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:22.614 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:22.614 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:22.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install libcephfs2 2023-12-08T10:04:23.375 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:23.598 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:23.599 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:23.600 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:23.693 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:20 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:23.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2023-12-08T10:04:24.562 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:23 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:25.263 INFO:teuthology.orchestra.run.smithi023.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:25.503 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:25.519 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:25.519 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:25.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install libcephfs-devel 2023-12-08T10:04:26.019 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:26.240 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:26.242 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:26.242 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:26.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2023-12-08T10:04:26.497 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:23 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:27.206 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:26 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:28.283 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:04:28.284 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:28.285 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 24 k 2023-12-08T10:04:28.285 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 94 k 2023-12-08T10:04:28.285 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:28.341 INFO:teuthology.orchestra.run.smithi023.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 438 kB/s | 24 kB 00:00 2023-12-08T10:04:28.341 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:28.341 INFO:teuthology.orchestra.run.smithi023.stdout:Total 427 kB/s | 24 kB 00:00 2023-12-08T10:04:28.342 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:28.346 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:28.346 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:28.348 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:28.351 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:28.399 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:28.589 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:28.656 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:28.865 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:28.879 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:28.880 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:28.880 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:28.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2023-12-08T10:04:29.561 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:29.561 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:29.561 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:29.562 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:29.562 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:29.562 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:29.703 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install librados2 2023-12-08T10:04:29.839 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:28 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:30.597 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:27 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:31.303 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:31.520 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:31.520 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:31.520 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:31.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2023-12-08T10:04:32.146 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:32.385 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:32.386 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:32.386 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:32.489 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:31 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:32.493 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install librbd1 2023-12-08T10:04:33.386 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:30 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:33.934 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:34.164 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:34.165 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:34.165 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:34.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2023-12-08T10:04:34.942 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:35.130 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:34 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:35.182 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:35.183 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:35.184 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:35.284 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-rados 2023-12-08T10:04:36.191 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:33 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:36.566 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:36.790 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:36.791 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2023-12-08T10:04:36.791 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:36.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2023-12-08T10:04:37.738 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:37.749 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:36 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:37.977 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:37.978 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:37.978 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:38.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-rgw 2023-12-08T10:04:38.969 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:35 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:39.414 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:39.414 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:39.414 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:39.415 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 89 k 2023-12-08T10:04:39.416 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 298 k 2023-12-08T10:04:39.416 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:39.459 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2023-12-08T10:04:39.460 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:39.460 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.0 MB/s | 89 kB 00:00 2023-12-08T10:04:39.460 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:39.470 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:39.470 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:39.521 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:39.524 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:39.636 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:39.837 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:40.116 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:40.514 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:40.752 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:40.753 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:40.753 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:40.787 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:40.787 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:40.787 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:40.787 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:40.787 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:40.787 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:40.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-cephfs 2023-12-08T10:04:41.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2023-12-08T10:04:41.746 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:38 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:41.869 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:40 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:43.297 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:43.540 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:43.542 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:43.542 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:43.543 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:43.544 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:43.544 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:43.544 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:04:43.545 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:43.546 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.7 M 2023-12-08T10:04:43.546 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 18 M 2023-12-08T10:04:43.546 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:43.631 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install python3-rbd 2023-12-08T10:04:43.645 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2023-12-08T10:04:43.646 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:43.646 INFO:teuthology.orchestra.run.smithi170.stdout:Total 37 MB/s | 3.7 MB 00:00 2023-12-08T10:04:43.647 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:43.658 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:43.659 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:43.722 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:43.725 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:44.193 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:44.396 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:44.527 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:41 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:44.842 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:45.499 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:45.499 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:45.500 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:45.500 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:45.500 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:45.500 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:45.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2023-12-08T10:04:46.092 INFO:teuthology.orchestra.run.smithi023.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-08T10:04:46.332 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:46.334 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2023-12-08T10:04:46.334 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:46.440 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install rbd-fuse 2023-12-08T10:04:46.559 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:02:45 ago on Fri 08 Dec 2023 10:02:01 AM UTC. 2023-12-08T10:04:47.333 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:44 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:48.250 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2023-12-08T10:04:48.251 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:48.252 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 173 k 2023-12-08T10:04:48.252 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 649 k 2023-12-08T10:04:48.252 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2023-12-08T10:04:48.298 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2023-12-08T10:04:48.299 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:48.299 INFO:teuthology.orchestra.run.smithi170.stdout:Total 3.6 MB/s | 173 kB 00:00 2023-12-08T10:04:48.299 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2023-12-08T10:04:48.310 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2023-12-08T10:04:48.310 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2023-12-08T10:04:48.359 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2023-12-08T10:04:48.362 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2023-12-08T10:04:48.476 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2023-12-08T10:04:48.682 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:48.972 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:49.135 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:04:49.136 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:49.137 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 89 k 2023-12-08T10:04:49.137 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 298 k 2023-12-08T10:04:49.137 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:49.230 INFO:teuthology.orchestra.run.smithi023.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 956 kB/s | 89 kB 00:00 2023-12-08T10:04:49.231 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:49.231 INFO:teuthology.orchestra.run.smithi023.stdout:Total 942 kB/s | 89 kB 00:00 2023-12-08T10:04:49.231 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:49.242 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:49.296 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:49.299 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:49.553 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:49.645 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:49.645 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2023-12-08T10:04:49.645 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:49.645 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:04:49.645 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2023-12-08T10:04:49.744 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:49.869 DEBUG:teuthology.parallel:result is None 2023-12-08T10:04:50.044 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:50.767 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:50.767 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:50.767 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:50.965 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install rbd-mirror 2023-12-08T10:04:51.874 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:48 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:53.695 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:04:53.696 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:53.697 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 3.7 M 2023-12-08T10:04:53.697 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 18 M 2023-12-08T10:04:53.697 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:53.782 INFO:teuthology.orchestra.run.smithi023.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-12-08T10:04:53.783 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:53.783 INFO:teuthology.orchestra.run.smithi023.stdout:Total 43 MB/s | 3.7 MB 00:00 2023-12-08T10:04:53.784 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:53.796 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:53.796 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:53.864 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:53.867 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:54.384 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:54.628 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:55.067 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:55.758 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:55.758 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:55.758 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:04:55.758 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-08T10:04:55.758 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:55.758 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:04:55.982 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install rbd-nbd 2023-12-08T10:04:56.875 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:02:53 ago on Fri 08 Dec 2023 10:02:03 AM UTC. 2023-12-08T10:04:58.659 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2023-12-08T10:04:58.659 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:58.659 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2023-12-08T10:04:58.659 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:58.659 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2023-12-08T10:04:58.659 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout:Install 1 Package 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 173 k 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 649 k 2023-12-08T10:04:58.660 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2023-12-08T10:04:58.702 INFO:teuthology.orchestra.run.smithi023.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.1 MB/s | 173 kB 00:00 2023-12-08T10:04:58.703 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2023-12-08T10:04:58.703 INFO:teuthology.orchestra.run.smithi023.stdout:Total 4.0 MB/s | 173 kB 00:00 2023-12-08T10:04:58.703 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2023-12-08T10:04:58.714 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2023-12-08T10:04:58.715 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2023-12-08T10:04:58.767 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2023-12-08T10:04:58.771 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2023-12-08T10:04:58.886 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2023-12-08T10:04:59.103 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:04:59.400 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:05:00.144 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-08T10:05:00.144 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:05:00.144 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2023-12-08T10:05:00.144 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-08T10:05:00.144 INFO:teuthology.orchestra.run.smithi023.stdout: 2023-12-08T10:05:00.144 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2023-12-08T10:05:00.325 DEBUG:teuthology.parallel:result is None 2023-12-08T10:05:00.327 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-08T10:05:00.327 INFO:teuthology.packaging:ref: None 2023-12-08T10:05:00.327 INFO:teuthology.packaging:tag: v16.2.4 2023-12-08T10:05:00.327 INFO:teuthology.packaging:branch: None 2023-12-08T10:05:00.327 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T10:05:00.542 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:05:00.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:05:00.761 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-08T10:05:00.786 INFO:teuthology.orchestra.run.smithi023.stdout:16.2.4-0.el8 2023-12-08T10:05:00.787 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-08T10:05:00.787 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-08T10:05:00.787 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-08T10:05:00.787 INFO:teuthology.packaging:ref: None 2023-12-08T10:05:00.788 INFO:teuthology.packaging:tag: v16.2.4 2023-12-08T10:05:00.788 INFO:teuthology.packaging:branch: None 2023-12-08T10:05:00.788 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-08T10:05:00.984 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:05:00.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-08T10:05:01.206 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-08T10:05:01.234 INFO:teuthology.orchestra.run.smithi170.stdout:16.2.4-0.el8 2023-12-08T10:05:01.234 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-08T10:05:01.234 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-08T10:05:01.235 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-08T10:05:01.235 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:05:01.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-08T10:05:01.270 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:05:01.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-08T10:05:01.308 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-08T10:05:01.308 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:05:01.308 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2023-12-08T10:05:01.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-08T10:05:01.412 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:05:01.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2023-12-08T10:05:01.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-08T10:05:01.511 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-08T10:05:01.511 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:05:01.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-08T10:05:01.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-08T10:05:01.611 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:05:01.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-08T10:05:01.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-08T10:05:01.708 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-08T10:05:01.709 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:05:01.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2023-12-08T10:05:01.738 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-08T10:05:01.808 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:05:01.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2023-12-08T10:05:01.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-08T10:05:01.903 INFO:teuthology.run_tasks:Running task print... 2023-12-08T10:05:01.913 INFO:teuthology.task.print:**** done install task... 2023-12-08T10:05:01.913 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-08T10:05:02.261 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'} 2023-12-08T10:05:02.261 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-12-08T10:05:02.261 INFO:tasks.cephadm:Cluster fsid is 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:02.262 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-08T10:05:02.262 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-08T10:05:02.262 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi023': '172.21.15.23', 'mon.smithi170': '172.21.15.170'} 2023-12-08T10:05:02.262 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-08T10:05:02.262 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2023-12-08T10:05:02.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2023-12-08T10:05:02.325 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-12-08T10:05:02.326 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-08T10:05:02.602 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 8 10:05 /home/ubuntu/cephtest/cephadm 2023-12-08T10:05:02.602 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-08T10:05:02.755 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 8 10:05 /home/ubuntu/cephtest/cephadm 2023-12-08T10:05:02.755 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-08T10:05:02.779 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-08T10:05:02.810 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2023-12-08T10:05:02.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2023-12-08T10:05:02.882 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2023-12-08T10:05:02.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2023-12-08T10:05:02.953 INFO:tasks.cephadm:Writing seed config... 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-08T10:05:02.954 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-08T10:05:02.955 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-08T10:05:02.956 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-08T10:05:02.957 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-08T10:05:02.957 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-08T10:05:02.957 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-08T10:05:02.957 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:05:02.957 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-08T10:05:02.984 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 40b80ed0-95b1-11ee-95a3-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-08T10:05:02.985 DEBUG:teuthology.orchestra.run.smithi023:mon.smithi023> sudo journalctl -f -n 0 -u ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023.service 2023-12-08T10:05:03.027 INFO:tasks.cephadm:Bootstrapping... 2023-12-08T10:05:03.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.23 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-08T10:05:03.056 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:-- Logs begin at Fri 2023-12-08 09:53:09 UTC. -- 2023-12-08T10:05:03.302 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-08T10:05:04.169 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 4.6.1 2023-12-08T10:05:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying podman|docker is present... 2023-12-08T10:05:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-08T10:05:04.204 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 4.6.1 2023-12-08T10:05:04.215 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying lvm2 is present... 2023-12-08T10:05:04.215 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying time synchronization is in place... 2023-12-08T10:05:04.215 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chrony.service 2023-12-08T10:05:04.226 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-08T10:05:04.226 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chrony.service 2023-12-08T10:05:04.229 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2023-12-08T10:05:04.230 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-08T10:05:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: enabled 2023-12-08T10:05:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chronyd.service 2023-12-08T10:05:04.243 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: active 2023-12-08T10:05:04.244 INFO:teuthology.orchestra.run.smithi023.stderr:Unit chronyd.service is enabled and running 2023-12-08T10:05:04.244 INFO:teuthology.orchestra.run.smithi023.stderr:Repeating the final host check... 2023-12-08T10:05:04.244 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-08T10:05:04.266 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 4.6.1 2023-12-08T10:05:04.278 INFO:teuthology.orchestra.run.smithi023.stderr:podman|docker (/bin/podman) is present 2023-12-08T10:05:04.279 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl is present 2023-12-08T10:05:04.279 INFO:teuthology.orchestra.run.smithi023.stderr:lvcreate is present 2023-12-08T10:05:04.279 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chrony.service 2023-12-08T10:05:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-08T10:05:04.285 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chrony.service 2023-12-08T10:05:04.291 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2023-12-08T10:05:04.292 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-08T10:05:04.297 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: enabled 2023-12-08T10:05:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active chronyd.service 2023-12-08T10:05:04.304 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: active 2023-12-08T10:05:04.304 INFO:teuthology.orchestra.run.smithi023.stderr:Unit chronyd.service is enabled and running 2023-12-08T10:05:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:Host looks OK 2023-12-08T10:05:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:Cluster fsid: 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:Acquiring lock 140387477441000 on /run/cephadm/40b80ed0-95b1-11ee-95a3-87774f69a715.lock 2023-12-08T10:05:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:Lock 140387477441000 acquired on /run/cephadm/40b80ed0-95b1-11ee-95a3-87774f69a715.lock 2023-12-08T10:05:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying IP 172.21.15.23 port 3300 ... 2023-12-08T10:05:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying IP 172.21.15.23 port 6789 ... 2023-12-08T10:05:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:Base mon IP is 172.21.15.23, final addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2023-12-08T10:05:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /sbin/ip route ls 2023-12-08T10:05:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.23 metric 100 2023-12-08T10:05:04.308 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2023-12-08T10:05:04.309 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /sbin/ip -6 route ls 2023-12-08T10:05:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-08T10:05:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-12-08T10:05:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1773sec hoplimit 64 pref medium 2023-12-08T10:05:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /sbin/ip -6 addr ls 2023-12-08T10:05:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-08T10:05:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-08T10:05:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-08T10:05:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-08T10:05:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2023-12-08T10:05:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-08T10:05:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:Mon IP 172.21.15.23 is in CIDR network 172.21.0.0/20 2023-12-08T10:05:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-08T10:05:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-12-08T10:05:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-12-08T10:05:04.404 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-12-08T10:05:10.965 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Getting image source signatures 2023-12-08T10:05:10.976 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-08T10:05:10.976 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-12-08T10:05:23.557 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-08T10:05:23.615 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: Writing manifest to image destination 2023-12-08T10:05:23.866 INFO:teuthology.orchestra.run.smithi023.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-08T10:05:23.882 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-12-08T10:05:24.417 INFO:teuthology.orchestra.run.smithi023.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-08T10:05:25.029 INFO:teuthology.orchestra.run.smithi023.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-08T10:05:25.029 INFO:teuthology.orchestra.run.smithi023.stderr:Extracting ceph user uid/gid from container image... 2023-12-08T10:05:25.029 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-12-08T10:05:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:stat: 167 167 2023-12-08T10:05:25.624 INFO:teuthology.orchestra.run.smithi023.stderr:Creating initial keys... 2023-12-08T10:05:25.624 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-08T10:05:25.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-authtool: AQBl6nJl2SYUNBAAC/N2c4soX6K9FEdo734fMA== 2023-12-08T10:05:26.239 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-08T10:05:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-authtool: AQBm6nJlu/mJHBAA0CvgIVkUZWhNNGeFBRdv9Q== 2023-12-08T10:05:26.813 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-08T10:05:27.063 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-authtool: AQBn6nJlW7ypAxAA8upYn9hZogDQM7ZgEDp78g== 2023-12-08T10:05:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:Creating initial monmap... 2023-12-08T10:05:27.407 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp49xa67j0:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 --addv smithi023 [v2:172.21.15.23:3300,v1:172.21.15.23:6789] /tmp/monmap 2023-12-08T10:05:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-08T10:05:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:27.680 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-08T10:05:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:monmaptool for smithi023 [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-08T10:05:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: set fsid to 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-08T10:05:28.025 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:05:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:Creating mon... 2023-12-08T10:05:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z -v /tmp/ceph-tmpvhvo6usf:/tmp/keyring:z -v /tmp/ceph-tmp49xa67j0:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi023 --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-12-08T10:05:28.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.288+0000 7f57874db700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.289+0000 7f57874db700 1 imported monmap: 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: last_changed 2023-12-08T10:05:27.678553+0000 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: created 2023-12-08T10:05:27.678553+0000 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.smithi023 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.289+0000 7f57874db700 0 /usr/bin/ceph-mon: set fsid to 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: RocksDB version: 6.8.1 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Compile date May 13 2021 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: DB SUMMARY 2023-12-08T10:05:28.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi023/store.db dir, Total Num: 0, files: 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi023/store.db: 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.error_if_exists: 0 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.create_if_missing: 1 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.paranoid_checks: 1 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.env: 0x55d60b2ec1c0 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.fs: Posix File System 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.info_log: 0x55d60bcaaec0 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.statistics: (nil) 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.use_fsync: 0 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_log_file_size: 0 2023-12-08T10:05:28.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.allow_fallocate: 1 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.use_direct_reads: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.db_log_dir: 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi023/store.db 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_subcompactions: 1 2023-12-08T10:05:28.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_background_flushes: -1 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.write_buffer_manager: 0x55d60bcb1530 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-08T10:05:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.unordered_write: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.row_cache: None 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.wal_filter: None 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.preserve_deletes: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.two_write_queues: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.atomic_flush: 0 2023-12-08T10:05:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.log_readahead_size: 0 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_background_jobs: 2 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_background_compactions: -1 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-08T10:05:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.max_open_files: -1 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Compression algorithms supported: 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kZSTD supported: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kXpressCompression supported: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kLZ4Compression supported: 1 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kBZip2Compression supported: 0 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kZlibCompression supported: 1 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: kSnappyCompression supported: 1 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.290+0000 7f57874db700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.294+0000 7f57874db700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-08T10:05:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000001 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.merge_operator: 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_filter: None 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d60bbc2120) 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-08T10:05:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache: 0x55d60bbf8d10 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-08T10:05:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-08T10:05:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression: NoCompression 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.num_levels: 7 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-08T10:05:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-08T10:05:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-08T10:05:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.table_properties_collectors: 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.inplace_update_support: 0 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.bloom_locality: 0 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.max_successive_merges: 0 2023-12-08T10:05:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.ttl: 2592000 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.295+0000 7f57874db700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.296+0000 7f57874db700 4 rocksdb: DB pointer 0x55d60bcbd800 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.296+0000 7f5770374700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.296+0000 7f5770374700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-08T10:05:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: 2023-12-08T10:05:28.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.298+0000 7f57874db700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-08T10:05:28.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.298+0000 7f57874db700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-08T10:05:28.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph-mon: debug 2023-12-08T10:05:28.298+0000 7f57874db700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi023 for mon.smithi023 2023-12-08T10:05:28.639 INFO:teuthology.orchestra.run.smithi023.stderr:create mon.smithi023 on 2023-12-08T10:05:28.640 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:28.644 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl disable ceph.target 2023-12-08T10:05:28.651 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-08T10:05:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph.target 2023-12-08T10:05:28.749 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-08T10:05:28.834 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph.target 2023-12-08T10:05:28.841 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target 2023-12-08T10:05:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target → /etc/systemd/system/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target. 2023-12-08T10:05:28.847 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target → /etc/systemd/system/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target. 2023-12-08T10:05:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target 2023-12-08T10:05:28.938 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl daemon-reload 2023-12-08T10:05:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023 2023-12-08T10:05:29.055 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl reset-failed ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023 2023-12-08T10:05:29.060 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to reset failed state of unit ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023.service: Unit ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023.service not loaded. 2023-12-08T10:05:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023 2023-12-08T10:05:29.067 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target.wants/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023.service → /etc/systemd/system/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@.service. 2023-12-08T10:05:29.157 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023 2023-12-08T10:05:29.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:29 smithi023 systemd[1]: Starting Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715... 2023-12-08T10:05:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-08T10:05:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2023-12-08T10:05:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2023-12-08T10:05:29.492 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2023-12-08T10:05:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2023-12-08T10:05:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to enable service . firewalld.service is not available 2023-12-08T10:05:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-08T10:05:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mon to start... 2023-12-08T10:05:29.494 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mon... 2023-12-08T10:05:29.494 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-12-08T10:05:29.863 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.859+0000 7fb303519700 1 Processor -- start 2023-12-08T10:05:29.863 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb303519700 1 -- start start 2023-12-08T10:05:29.863 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb303519700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb303519700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0fedd0 con 0x7fb2fc070430 2023-12-08T10:05:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb3012b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb3012b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41498/0 (socket says 172.21.15.23:41498) 2023-12-08T10:05:29.866 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb3012b5700 1 -- 172.21.15.23:0/2425515289 learned_addr learned my addr 172.21.15.23:0/2425515289 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.860+0000 7fb3012b5700 1 -- 172.21.15.23:0/2425515289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0fef10 con 0x7fb2fc070430 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb3012b5700 1 --2- 172.21.15.23:0/2425515289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc0fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb2f800b5f0 tx=0x7fb2f8006ee0).ready entity=mon.0 client_cookie=f3fd4360a0b3a7cd server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb2effff700 1 -- 172.21.15.23:0/2425515289 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f800d600 con 0x7fb2fc070430 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb2effff700 1 -- 172.21.15.23:0/2425515289 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f800d760 con 0x7fb2fc070430 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb2effff700 1 -- 172.21.15.23:0/2425515289 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f800da30 con 0x7fb2fc070430 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb303519700 1 -- 172.21.15.23:0/2425515289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 msgr2=0x7fb2fc0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb303519700 1 --2- 172.21.15.23:0/2425515289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc0fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb2f800b5f0 tx=0x7fb2f8006ee0).stop 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb303519700 1 -- 172.21.15.23:0/2425515289 shutdown_connections 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb303519700 1 --2- 172.21.15.23:0/2425515289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc0fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb303519700 1 -- 172.21.15.23:0/2425515289 >> 172.21.15.23:0/2425515289 conn(0x7fb2fc0f63c0 msgr2=0x7fb2fc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.861+0000 7fb303519700 1 -- 172.21.15.23:0/2425515289 shutdown_connections 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.862+0000 7fb303519700 1 -- 172.21.15.23:0/2425515289 wait complete. 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.862+0000 7fb303519700 1 Processor -- start 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.862+0000 7fb303519700 1 -- start start 2023-12-08T10:05:29.867 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.862+0000 7fb303519700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc10a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.862+0000 7fb303519700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc10a770 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb3012b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc10a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb3012b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc10a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41504/0 (socket says 172.21.15.23:41504) 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb3012b5700 1 -- 172.21.15.23:0/4080631685 learned_addr learned my addr 172.21.15.23:0/4080631685 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb3012b5700 1 -- 172.21.15.23:0/4080631685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f800b200 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb3012b5700 1 --2- 172.21.15.23:0/4080631685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc10a280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb2f8006c70 tx=0x7fb2f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f8011b40 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8011ca0 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc10a970 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f801d430 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.863+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc10ad80 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.864+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb2f800f070 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.864+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb2f8027460 con 0x7fb2fc070430 2023-12-08T10:05:29.868 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.864+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc045ad0 con 0x7fb2fc070430 2023-12-08T10:05:29.869 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.867+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb2f8016070 con 0x7fb2fc070430 2023-12-08T10:05:29.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:29 smithi023 bash[111316]: 54c23ffecd14933583f4b94ff1e61f158280943d4a5244cc25242adfaf86b710 2023-12-08T10:05:29.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:29 smithi023 systemd[1]: Started Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715. 2023-12-08T10:05:29.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.507389+0000 mon.smithi023 (mon.0) 0 : [INF] mkfs 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:29.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.491959+0000 mon.smithi023 (mon.0) 1 : cluster [INF] mon.smithi023 is new leader, mons smithi023 in quorum (ranks 0) 2023-12-08T10:05:29.949 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.947+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb2fc103dc0 con 0x7fb2fc070430 2023-12-08T10:05:29.951 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.947+0000 7fb2edffb700 1 -- 172.21.15.23:0/4080631685 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fb2f8036070 con 0x7fb2fc070430 2023-12-08T10:05:29.951 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: cluster: 2023-12-08T10:05:29.951 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: id: 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:29.951 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-08T10:05:29.951 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: services: 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi023 (age 0.442178s) 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: data: 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-08T10:05:29.952 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: pgs: 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 msgr2=0x7fb2fc10a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 --2- 172.21.15.23:0/4080631685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc10a280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb2f8006c70 tx=0x7fb2f8016000).stop 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 shutdown_connections 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 --2- 172.21.15.23:0/4080631685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2fc070430 0x7fb2fc10a280 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 >> 172.21.15.23:0/4080631685 conn(0x7fb2fc0f63c0 msgr2=0x7fb2fc0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 shutdown_connections 2023-12-08T10:05:29.953 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:29.948+0000 7fb303519700 1 -- 172.21.15.23:0/4080631685 wait complete. 2023-12-08T10:05:30.282 INFO:teuthology.orchestra.run.smithi023.stderr:mon is available 2023-12-08T10:05:30.282 INFO:teuthology.orchestra.run.smithi023.stderr:Assimilating anything we can from ceph.conf... 2023-12-08T10:05:30.282 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z -v /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi023/config 2023-12-08T10:05:30.665 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.662+0000 7f3214efe700 1 Processor -- start 2023-12-08T10:05:30.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.662+0000 7f3214efe700 1 -- start start 2023-12-08T10:05:30.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f3214efe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f3210070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:30.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f3214efe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100728b0 con 0x7f3210072490 2023-12-08T10:05:30.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f3210070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:30.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f3210070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41516/0 (socket says 172.21.15.23:41516) 2023-12-08T10:05:30.666 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320e59c700 1 -- 172.21.15.23:0/3688840054 learned_addr learned my addr 172.21.15.23:0/3688840054 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:30.669 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320e59c700 1 -- 172.21.15.23:0/3688840054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210070f20 con 0x7f3210072490 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320e59c700 1 --2- 172.21.15.23:0/3688840054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f3210070a30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f320000b5f0 tx=0x7f3200006ee0).ready entity=mon.0 client_cookie=7afa61a62c5a3405 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320d59a700 1 -- 172.21.15.23:0/3688840054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f320000d600 con 0x7f3210072490 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320d59a700 1 -- 172.21.15.23:0/3688840054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320000d760 con 0x7f3210072490 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.663+0000 7f320d59a700 1 -- 172.21.15.23:0/3688840054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f320000da30 con 0x7f3210072490 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 -- 172.21.15.23:0/3688840054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 msgr2=0x7f3210070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 --2- 172.21.15.23:0/3688840054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f3210070a30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f320000b5f0 tx=0x7f3200006ee0).stop 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 -- 172.21.15.23:0/3688840054 shutdown_connections 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 --2- 172.21.15.23:0/3688840054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f3210070a30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 -- 172.21.15.23:0/3688840054 >> 172.21.15.23:0/3688840054 conn(0x7f32100f74c0 msgr2=0x7f32100f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 -- 172.21.15.23:0/3688840054 shutdown_connections 2023-12-08T10:05:30.670 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.664+0000 7f3214efe700 1 -- 172.21.15.23:0/3688840054 wait complete. 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.665+0000 7f3214efe700 1 Processor -- start 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.665+0000 7f3214efe700 1 -- start start 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.665+0000 7f3214efe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f321010c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.665+0000 7f3214efe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321010c9b0 con 0x7f3210072490 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f320e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f321010c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f320e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f321010c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41532/0 (socket says 172.21.15.23:41532) 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f320e59c700 1 -- 172.21.15.23:0/2462322186 learned_addr learned my addr 172.21.15.23:0/2462322186 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f320e59c700 1 -- 172.21.15.23:0/2462322186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320000b200 con 0x7f3210072490 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f320e59c700 1 --2- 172.21.15.23:0/2462322186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f321010c4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3200006c70 tx=0x7f3200016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3200011b40 con 0x7f3210072490 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3200011ca0 con 0x7f3210072490 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321010cbb0 con 0x7f3210072490 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f320001d430 con 0x7f3210072490 2023-12-08T10:05:30.671 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.666+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321010cfc0 con 0x7f3210072490 2023-12-08T10:05:30.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.667+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f320000f070 con 0x7f3210072490 2023-12-08T10:05:30.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.667+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3200027460 con 0x7f3210072490 2023-12-08T10:05:30.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.667+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3210045ad0 con 0x7f3210072490 2023-12-08T10:05:30.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.670+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3200016070 con 0x7f3210072490 2023-12-08T10:05:30.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.506679+0000 mon.smithi023 (mon.0) 2 : cluster [INF] mon.smithi023 is new leader, mons smithi023 in quorum (ranks 0) 2023-12-08T10:05:30.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.506931+0000 mon.smithi023 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi023=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} 2023-12-08T10:05:30.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.508903+0000 mon.smithi023 (mon.0) 4 : cluster [DBG] fsmap 2023-12-08T10:05:30.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.515781+0000 mon.smithi023 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-08T10:05:30.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: cluster 2023-12-08T10:05:29.519571+0000 mon.smithi023 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-08T10:05:30.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: audit 2023-12-08T10:05:29.948815+0000 mon.smithi023 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.23:0/4080631685' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-08T10:05:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.737+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3210071290 con 0x7f3210072490 2023-12-08T10:05:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.745+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f320001d5e0 con 0x7f3210072490 2023-12-08T10:05:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.745+0000 7f31feffd700 1 -- 172.21.15.23:0/2462322186 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7f3200036020 con 0x7f3210072490 2023-12-08T10:05:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: [global] 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: fsid = 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: [mgr] 2023-12-08T10:05:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-08T10:05:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: [osd] 2023-12-08T10:05:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-08T10:05:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-08T10:05:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 msgr2=0x7f321010c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 --2- 172.21.15.23:0/2462322186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f321010c4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f3200006c70 tx=0x7f3200016000).stop 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 shutdown_connections 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 --2- 172.21.15.23:0/2462322186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3210072490 0x7f321010c4c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 >> 172.21.15.23:0/2462322186 conn(0x7f32100f74c0 msgr2=0x7f3210100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 shutdown_connections 2023-12-08T10:05:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:30.746+0000 7f3214efe700 1 -- 172.21.15.23:0/2462322186 wait complete. 2023-12-08T10:05:31.089 INFO:teuthology.orchestra.run.smithi023.stderr:Generating new minimal ceph.conf... 2023-12-08T10:05:31.089 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z -v /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/mon.smithi023:/var/lib/ceph/mon/ceph-smithi023:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi023/config 2023-12-08T10:05:31.444 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.441+0000 7f88758a5700 1 Processor -- start 2023-12-08T10:05:31.444 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.441+0000 7f88758a5700 1 -- start start 2023-12-08T10:05:31.444 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.441+0000 7f88758a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f88700fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:31.444 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.441+0000 7f88758a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700fb490 con 0x7f88700fab80 2023-12-08T10:05:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f88700fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f88700fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41540/0 (socket says 172.21.15.23:41540) 2023-12-08T10:05:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886effd700 1 -- 172.21.15.23:0/2154201992 learned_addr learned my addr 172.21.15.23:0/2154201992 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886effd700 1 -- 172.21.15.23:0/2154201992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700fb5d0 con 0x7f88700fab80 2023-12-08T10:05:31.446 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886effd700 1 --2- 172.21.15.23:0/2154201992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f88700fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f886400b870 tx=0x7f8864006f70).ready entity=mon.0 client_cookie=5b7f67cda4f3d000 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886dffb700 1 -- 172.21.15.23:0/2154201992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886400da20 con 0x7f88700fab80 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886dffb700 1 -- 172.21.15.23:0/2154201992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8864010470 con 0x7f88700fab80 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.442+0000 7f886dffb700 1 -- 172.21.15.23:0/2154201992 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886400f5b0 con 0x7f88700fab80 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 -- 172.21.15.23:0/2154201992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 msgr2=0x7f88700fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 --2- 172.21.15.23:0/2154201992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f88700fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f886400b870 tx=0x7f8864006f70).stop 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 -- 172.21.15.23:0/2154201992 shutdown_connections 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 --2- 172.21.15.23:0/2154201992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f88700fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 -- 172.21.15.23:0/2154201992 >> 172.21.15.23:0/2154201992 conn(0x7f88700f63c0 msgr2=0x7f88700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 -- 172.21.15.23:0/2154201992 shutdown_connections 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.443+0000 7f88758a5700 1 -- 172.21.15.23:0/2154201992 wait complete. 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f88758a5700 1 Processor -- start 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f88758a5700 1 -- start start 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f88758a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f8870108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f88758a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870108c60 con 0x7f88700fab80 2023-12-08T10:05:31.447 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f886effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f8870108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f886effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f8870108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41550/0 (socket says 172.21.15.23:41550) 2023-12-08T10:05:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.444+0000 7f886effd700 1 -- 172.21.15.23:0/3203348423 learned_addr learned my addr 172.21.15.23:0/3203348423 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f886effd700 1 -- 172.21.15.23:0/3203348423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886400b480 con 0x7f88700fab80 2023-12-08T10:05:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f886effd700 1 --2- 172.21.15.23:0/3203348423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f8870108770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8864006a40 tx=0x7f8864006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f886400f3d0 con 0x7f88700fab80 2023-12-08T10:05:31.448 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f886400fef0 con 0x7f88700fab80 2023-12-08T10:05:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8870108e60 con 0x7f88700fab80 2023-12-08T10:05:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8864019d40 con 0x7f88700fab80 2023-12-08T10:05:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8870109270 con 0x7f88700fab80 2023-12-08T10:05:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f886402b060 con 0x7f88700fab80 2023-12-08T10:05:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.445+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f886402ea60 con 0x7f88700fab80 2023-12-08T10:05:31.449 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.447+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870045ad0 con 0x7f88700fab80 2023-12-08T10:05:31.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.449+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8864029070 con 0x7f88700fab80 2023-12-08T10:05:31.519 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.517+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8870102670 con 0x7f88700fab80 2023-12-08T10:05:31.519 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.517+0000 7f88577fe700 1 -- 172.21.15.23:0/3203348423 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f8864014070 con 0x7f88700fab80 2023-12-08T10:05:31.520 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 msgr2=0x7f8870108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:31.520 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 --2- 172.21.15.23:0/3203348423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f8870108770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8864006a40 tx=0x7f8864006f70).stop 2023-12-08T10:05:31.520 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 shutdown_connections 2023-12-08T10:05:31.520 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 --2- 172.21.15.23:0/3203348423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88700fab80 0x7f8870108770 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:31.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 >> 172.21.15.23:0/3203348423 conn(0x7f88700f63c0 msgr2=0x7f88700ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:31.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 shutdown_connections 2023-12-08T10:05:31.521 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:31.518+0000 7f88758a5700 1 -- 172.21.15.23:0/3203348423 wait complete. 2023-12-08T10:05:31.825 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: audit 2023-12-08T10:05:30.738650+0000 mon.smithi023 (mon.0) 8 : audit [INF] from='client.? 172.21.15.23:0/2462322186' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-08T10:05:31.825 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: audit 2023-12-08T10:05:30.746582+0000 mon.smithi023 (mon.0) 9 : audit [INF] from='client.? 172.21.15.23:0/2462322186' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-08T10:05:31.825 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: audit 2023-12-08T10:05:31.518429+0000 mon.smithi023 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.23:0/3203348423' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-08T10:05:31.882 INFO:teuthology.orchestra.run.smithi023.stderr:Restarting the monitor... 2023-12-08T10:05:31.882 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl restart ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023 2023-12-08T10:05:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:31 smithi023 systemd[1]: Stopping Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715... 2023-12-08T10:05:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: debug 2023-12-08T10:05:31.952+0000 7f90edc3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi023 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-08T10:05:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111330]: debug 2023-12-08T10:05:31.952+0000 7f90edc3f700 -1 mon.smithi023@0(leader) e1 *** Got Signal Terminated *** 2023-12-08T10:05:32.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 podman[111652]: ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023 2023-12-08T10:05:32.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 systemd[1]: ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023.service: Succeeded. 2023-12-08T10:05:32.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 systemd[1]: Stopped Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715. 2023-12-08T10:05:32.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 systemd[1]: Starting Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715... 2023-12-08T10:05:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-08T10:05:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-12-08T10:05:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.148+0000 7fa25759e700 1 Processor -- start 2023-12-08T10:05:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.148+0000 7fa25759e700 1 -- start start 2023-12-08T10:05:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.148+0000 7fa25759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.148+0000 7fa25759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580fec80 con 0x7fa258070460 2023-12-08T10:05:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41552/0 (socket says 172.21.15.23:41552) 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25659c700 1 -- 172.21.15.23:0/967578157 learned_addr learned my addr 172.21.15.23:0/967578157 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25659c700 1 -- 172.21.15.23:0/967578157 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2580fedc0 con 0x7fa258070460 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25659c700 1 --2- 172.21.15.23:0/967578157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fe790 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa24800b5f0 tx=0x7fa248006ee0).ready entity=mon.0 client_cookie=57715ac14836a0ba server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25559a700 1 -- 172.21.15.23:0/967578157 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24800d600 con 0x7fa258070460 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25559a700 1 -- 172.21.15.23:0/967578157 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa24800dbd0 con 0x7fa258070460 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.149+0000 7fa25559a700 1 -- 172.21.15.23:0/967578157 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa248011c00 con 0x7fa258070460 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 -- 172.21.15.23:0/967578157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 msgr2=0x7fa2580fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 --2- 172.21.15.23:0/967578157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fe790 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fa24800b5f0 tx=0x7fa248006ee0).stop 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 -- 172.21.15.23:0/967578157 shutdown_connections 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 --2- 172.21.15.23:0/967578157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fe790 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 -- 172.21.15.23:0/967578157 >> 172.21.15.23:0/967578157 conn(0x7fa2580f6250 msgr2=0x7fa2580f8670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 -- 172.21.15.23:0/967578157 shutdown_connections 2023-12-08T10:05:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.150+0000 7fa25759e700 1 -- 172.21.15.23:0/967578157 wait complete. 2023-12-08T10:05:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25759e700 1 Processor -- start 2023-12-08T10:05:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25759e700 1 -- start start 2023-12-08T10:05:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2580fa690 con 0x7fa258070460 2023-12-08T10:05:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41556/0 (socket says 172.21.15.23:41556) 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25659c700 1 -- 172.21.15.23:0/3248730930 learned_addr learned my addr 172.21.15.23:0/3248730930 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25659c700 1 -- 172.21.15.23:0/3248730930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24800b200 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.151+0000 7fa25659c700 1 --2- 172.21.15.23:0/3248730930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fc210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa24800b1d0 tx=0x7fa248006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa248011a20 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa24801b880 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2580fa830 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa24801a700 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2580fac40 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fa24800f070 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.152+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa2480259b0 con 0x7fa258070460 2023-12-08T10:05:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.153+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa258045ad0 con 0x7fa258070460 2023-12-08T10:05:33.157 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.155+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fa248016070 con 0x7fa258070460 2023-12-08T10:05:33.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 bash[111724]: 773834bb9762342456a28dc5cb57d44f7f54fec697fe73601532fa47390381a8 2023-12-08T10:05:33.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 systemd[1]: Started Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715. 2023-12-08T10:05:33.186 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.754+0000 7f192ea7b700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-08T10:05:33.187 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.754+0000 7f192ea7b700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-12-08T10:05:33.187 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.754+0000 7f192ea7b700 0 pidfile_write: ignore empty --pid-file 2023-12-08T10:05:33.187 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.758+0000 7f192ea7b700 0 load: jerasure load: lrc load: isa 2023-12-08T10:05:33.187 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: RocksDB version: 6.8.1 2023-12-08T10:05:33.187 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.187 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Compile date May 13 2021 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: DB SUMMARY 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: CURRENT file: CURRENT 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-08T10:05:33.188 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-08T10:05:33.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi023/store.db dir, Total Num: 1, files: 000004.sst 2023-12-08T10:05:33.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi023/store.db: 000006.log size: 93438 ; 2023-12-08T10:05:33.189 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.error_if_exists: 0 2023-12-08T10:05:33.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.create_if_missing: 0 2023-12-08T10:05:33.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.paranoid_checks: 1 2023-12-08T10:05:33.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.env: 0x5626873c11c0 2023-12-08T10:05:33.190 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.fs: Posix File System 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.info_log: 0x562688173960 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.statistics: (nil) 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.use_fsync: 0 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_log_file_size: 0 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-08T10:05:33.191 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.allow_fallocate: 1 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.use_direct_reads: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.db_log_dir: 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi023/store.db 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_subcompactions: 1 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_background_flushes: -1 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-08T10:05:33.192 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.write_buffer_manager: 0x562688de8030 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-08T10:05:33.193 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.unordered_write: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.row_cache: None 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.wal_filter: None 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.preserve_deletes: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.two_write_queues: 0 2023-12-08T10:05:33.194 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.atomic_flush: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.log_readahead_size: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_background_jobs: 2 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_background_compactions: -1 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-08T10:05:33.195 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_open_files: -1 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Compression algorithms supported: 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kZSTD supported: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kXpressCompression supported: 0 2023-12-08T10:05:33.196 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kLZ4Compression supported: 1 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kBZip2Compression supported: 0 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kZlibCompression supported: 1 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: kSnappyCompression supported: 1 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000005 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.merge_operator: 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_filter: None 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-08T10:05:33.197 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56268808a128) 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cache_index_and_filter_blocks: 1 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: pin_top_level_index_and_filter: 1 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: index_type: 0 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: data_block_index_type: 0 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: index_shortening: 1 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: data_block_hash_table_util_ratio: 0.750000 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: hash_index_allow_collision: 1 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: checksum: 1 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: no_block_cache: 0 2023-12-08T10:05:33.198 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_cache: 0x5626880c0f10 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_cache_name: BinnedLRUCache 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_cache_options: 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: capacity : 536870912 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: num_shard_bits : 4 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: strict_capacity_limit : 0 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: high_pri_pool_ratio: 0.000 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_cache_compressed: (nil) 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: persistent_cache: (nil) 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_size: 4096 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_size_deviation: 10 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_restart_interval: 16 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: index_block_restart_interval: 1 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: metadata_block_size: 4096 2023-12-08T10:05:33.199 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: partition_filters: 0 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: use_delta_encoding: 1 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: whole_key_filtering: 1 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: verify_compression: 0 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: read_amp_bytes_per_bit: 0 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: format_version: 2 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: enable_index_compression: 1 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: block_align: 0 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression: NoCompression 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-08T10:05:33.200 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.num_levels: 7 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-08T10:05:33.201 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-08T10:05:33.202 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-08T10:05:33.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-08T10:05:33.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-08T10:05:33.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-08T10:05:33.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-08T10:05:33.203 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-08T10:05:33.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-08T10:05:33.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-08T10:05:33.205 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-08T10:05:33.206 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.table_properties_collectors: 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.inplace_update_support: 0 2023-12-08T10:05:33.207 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.bloom_locality: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.max_successive_merges: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.ttl: 2592000 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.759+0000 7f192ea7b700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.760+0000 7f192ea7b700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi023/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.760+0000 7f192ea7b700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-08T10:05:33.208 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.760+0000 7f192ea7b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702029932761982, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.760+0000 7f192ea7b700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.761+0000 7f192ea7b700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.762+0000 7f192ea7b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702029932763464, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 89218, "table_properties": {"data_size": 87333, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 80474, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702029932, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.762+0000 7f192ea7b700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.763+0000 7f192ea7b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702029932764232, "job": 1, "event": "recovery_finished"} 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.764+0000 7f192ea7b700 4 rocksdb: DB pointer 0x562688185800 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.764+0000 7f1915893700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.764+0000 7f1915893700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** DB Stats ** 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-08T10:05:33.209 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** Compaction Stats [default] ** 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: L0 2/0 88.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.210 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Sum 2/0 88.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** Compaction Stats [default] ** 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(Total Files): cumulative 0, interval 0 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(Keys): cumulative 0, interval 0 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Cumulative compaction: 0.00 GB write, 19.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Interval compaction: 0.00 GB write, 19.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:33.211 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** File Read Latency Histogram By Level [default] ** 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** Compaction Stats [default] ** 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: L0 2/0 88.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Sum 2/0 88.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** Compaction Stats [default] ** 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 85.8 0.00 0.00 1 0.001 0 0 2023-12-08T10:05:33.212 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(Total Files): cumulative 0, interval 0 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: AddFile(Keys): cumulative 0, interval 0 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Cumulative compaction: 0.00 GB write, 18.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ** File Read Latency Histogram By Level [default] ** 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.213 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.765+0000 7f192ea7b700 0 starting mon.smithi023 rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi023 fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.766+0000 7f192ea7b700 1 mon.smithi023@-1(???) e1 preinit fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 0 mon.smithi023@-1(???).mds e1 new map 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 0 mon.smithi023@-1(???).mds e1 print_map 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: e1 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: legacy client fscid: -1 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: No filesystems configured 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 0 mon.smithi023@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 0 mon.smithi023@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-08T10:05:33.214 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 0 mon.smithi023@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 0 mon.smithi023@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:30.738650+0000 mon.smithi023 (mon.0) 8 : audit [INF] from='client.? 172.21.15.23:0/2462322186' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:30.746582+0000 mon.smithi023 (mon.0) 9 : audit [INF] from='client.? 172.21.15.23:0/2462322186' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:31.518429+0000 mon.smithi023 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.23:0/3203348423' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.767+0000 7f192ea7b700 1 mon.smithi023@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expand map: {default=false} 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta from 'false' to 'false' 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expanded map: {default=false} 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expand map: {default=info} 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta from 'info' to 'info' 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expanded map: {default=info} 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expand map: {default=daemon} 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expanded map: {default=daemon} 2023-12-08T10:05:33.215 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expand map: {default=debug} 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta from 'debug' to 'debug' 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:05:32.769+0000 7f192ea7b700 20 expand_channel_meta expanded map: {default=debug} 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:32.785873+0000 mon.smithi023 (mon.0) 1 : cluster [INF] mon.smithi023 is new leader, mons smithi023 in quorum (ranks 0) 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:32.785933+0000 mon.smithi023 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi023=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:32.785975+0000 mon.smithi023 (mon.0) 3 : cluster [DBG] fsmap 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:32.786002+0000 mon.smithi023 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-08T10:05:33.216 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:32.786441+0000 mon.smithi023 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-08T10:05:33.225 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.222+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fa258104700 con 0x7fa258070460 2023-12-08T10:05:33.232 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.230+0000 7fa246ffd700 1 -- 172.21.15.23:0/3248730930 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa24801bae0 con 0x7fa258070460 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 msgr2=0x7fa2580fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 --2- 172.21.15.23:0/3248730930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fc210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa24800b1d0 tx=0x7fa248006ee0).stop 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 shutdown_connections 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 --2- 172.21.15.23:0/3248730930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa258070460 0x7fa2580fc210 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 >> 172.21.15.23:0/3248730930 conn(0x7fa2580f6250 msgr2=0x7fa2580f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 shutdown_connections 2023-12-08T10:05:33.235 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:33.231+0000 7fa25759e700 1 -- 172.21.15.23:0/3248730930 wait complete. 2023-12-08T10:05:33.582 INFO:teuthology.orchestra.run.smithi023.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-08T10:05:33.582 INFO:teuthology.orchestra.run.smithi023.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-08T10:05:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:Creating mgr... 2023-12-08T10:05:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:Verifying port 9283 ... 2023-12-08T10:05:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl daemon-reload 2023-12-08T10:05:33.685 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg 2023-12-08T10:05:33.712 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl reset-failed ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg 2023-12-08T10:05:33.718 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Failed to reset failed state of unit ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg.service: Unit ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg.service not loaded. 2023-12-08T10:05:33.719 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl enable ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg 2023-12-08T10:05:33.724 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: Created symlink /etc/systemd/system/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715.target.wants/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg.service → /etc/systemd/system/ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@.service. 2023-12-08T10:05:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mgr.smithi023.tgiigg 2023-12-08T10:05:34.124 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-08T10:05:34.133 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2023-12-08T10:05:34.134 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2023-12-08T10:05:34.143 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2023-12-08T10:05:34.144 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2023-12-08T10:05:34.144 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to enable service . firewalld.service is not available 2023-12-08T10:05:34.144 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-08T10:05:34.144 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-08T10:05:34.153 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2023-12-08T10:05:34.154 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2023-12-08T10:05:34.162 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2023-12-08T10:05:34.163 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2023-12-08T10:05:34.164 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-08T10:05:34.164 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr to start... 2023-12-08T10:05:34.164 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr... 2023-12-08T10:05:34.164 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-08T10:05:34.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:34 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:33.231430+0000 mon.smithi023 (mon.0) 6 : audit [INF] from='client.? 172.21.15.23:0/3248730930' entity='client.admin' 2023-12-08T10:05:34.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.534+0000 7f3ca8c0f700 1 Processor -- start 2023-12-08T10:05:34.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.534+0000 7f3ca8c0f700 1 -- start start 2023-12-08T10:05:34.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.535+0000 7f3ca8c0f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:34.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.535+0000 7f3ca8c0f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca40fb490 con 0x7f3ca40fab80 2023-12-08T10:05:34.538 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.535+0000 7f3ca259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:34.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.535+0000 7f3ca259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41580/0 (socket says 172.21.15.23:41580) 2023-12-08T10:05:34.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.535+0000 7f3ca259c700 1 -- 172.21.15.23:0/3732167173 learned_addr learned my addr 172.21.15.23:0/3732167173 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:34.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.535+0000 7f3ca259c700 1 -- 172.21.15.23:0/3732167173 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca40fb5d0 con 0x7f3ca40fab80 2023-12-08T10:05:34.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca259c700 1 --2- 172.21.15.23:0/3732167173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca40fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3c9400b870 tx=0x7f3c94006f70).ready entity=mon.0 client_cookie=174e6db9ded23f53 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:34.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca159a700 1 -- 172.21.15.23:0/3732167173 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c9400da20 con 0x7f3ca40fab80 2023-12-08T10:05:34.540 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca159a700 1 -- 172.21.15.23:0/3732167173 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3c94010470 con 0x7f3ca40fab80 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca159a700 1 -- 172.21.15.23:0/3732167173 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c9400f5b0 con 0x7f3ca40fab80 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/3732167173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 msgr2=0x7f3ca40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca8c0f700 1 --2- 172.21.15.23:0/3732167173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca40fafa0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f3c9400b870 tx=0x7f3c94006f70).stop 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.536+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/3732167173 shutdown_connections 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.537+0000 7f3ca8c0f700 1 --2- 172.21.15.23:0/3732167173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca40fafa0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.537+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/3732167173 >> 172.21.15.23:0/3732167173 conn(0x7f3ca40f63c0 msgr2=0x7f3ca40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.537+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/3732167173 shutdown_connections 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.537+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/3732167173 wait complete. 2023-12-08T10:05:34.541 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.537+0000 7f3ca8c0f700 1 Processor -- start 2023-12-08T10:05:34.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.537+0000 7f3ca8c0f700 1 -- start start 2023-12-08T10:05:34.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca8c0f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca410cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:34.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca8c0f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca410d070 con 0x7f3ca40fab80 2023-12-08T10:05:34.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca410cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:34.542 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca410cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41582/0 (socket says 172.21.15.23:41582) 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca259c700 1 -- 172.21.15.23:0/1994438446 learned_addr learned my addr 172.21.15.23:0/1994438446 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca259c700 1 -- 172.21.15.23:0/1994438446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9400b480 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca259c700 1 --2- 172.21.15.23:0/1994438446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca410cb80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3c94006a40 tx=0x7f3c94006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c9400de00 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3c9400fef0 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca410d270 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c94019e90 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.538+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca410d680 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.539+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca4045ad0 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.539+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f3c9402a070 con 0x7f3ca40fab80 2023-12-08T10:05:34.543 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.539+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3c94022d40 con 0x7f3ca40fab80 2023-12-08T10:05:34.544 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.542+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3c94028060 con 0x7f3ca40fab80 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.612+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3ca40fc890 con 0x7f3ca40fab80 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.613+0000 7f3c92ffd700 1 -- 172.21.15.23:0/1994438446 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f3c94031020 con 0x7f3ca40fab80 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "40b80ed0-95b1-11ee-95a3-87774f69a715", 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2023-12-08T10:05:34.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2023-12-08T10:05:34.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-08T10:05:34.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2023-12-08T10:05:34.624 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": false, 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2023-12-08T10:05:29.507833+0000", 2023-12-08T10:05:34.625 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 msgr2=0x7f3ca410cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 --2- 172.21.15.23:0/1994438446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca410cb80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f3c94006a40 tx=0x7f3c94006f70).stop 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 shutdown_connections 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 --2- 172.21.15.23:0/1994438446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca40fab80 0x7f3ca410cb80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 >> 172.21.15.23:0/1994438446 conn(0x7f3ca40f63c0 msgr2=0x7f3ca40ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 shutdown_connections 2023-12-08T10:05:34.626 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:34.614+0000 7f3ca8c0f700 1 -- 172.21.15.23:0/1994438446 wait complete. 2023-12-08T10:05:34.981 INFO:teuthology.orchestra.run.smithi023.stderr:mgr not available, waiting (1/15)... 2023-12-08T10:05:35.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:34.614248+0000 mon.smithi023 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.23:0/1994438446' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-08T10:05:36.984 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.344+0000 7f5ce581d700 1 Processor -- start 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.344+0000 7f5ce581d700 1 -- start start 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.344+0000 7f5ce581d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.344+0000 7f5ce581d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00fee10 con 0x7f5ce00703f0 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.344+0000 7f5cdeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cdeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41594/0 (socket says 172.21.15.23:41594) 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cdeffd700 1 -- 172.21.15.23:0/372552282 learned_addr learned my addr 172.21.15.23:0/372552282 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cdeffd700 1 -- 172.21.15.23:0/372552282 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce00fef50 con 0x7f5ce00703f0 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cdeffd700 1 --2- 172.21.15.23:0/372552282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fe920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5cd000b5f0 tx=0x7f5cd0006ee0).ready entity=mon.0 client_cookie=97135da994893601 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cddffb700 1 -- 172.21.15.23:0/372552282 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd000d600 con 0x7f5ce00703f0 2023-12-08T10:05:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cddffb700 1 -- 172.21.15.23:0/372552282 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cd000dbd0 con 0x7f5ce00703f0 2023-12-08T10:05:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.345+0000 7f5cddffb700 1 -- 172.21.15.23:0/372552282 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd0011c00 con 0x7f5ce00703f0 2023-12-08T10:05:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 -- 172.21.15.23:0/372552282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 msgr2=0x7f5ce00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 --2- 172.21.15.23:0/372552282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fe920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5cd000b5f0 tx=0x7f5cd0006ee0).stop 2023-12-08T10:05:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 -- 172.21.15.23:0/372552282 shutdown_connections 2023-12-08T10:05:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 --2- 172.21.15.23:0/372552282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fe920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 -- 172.21.15.23:0/372552282 >> 172.21.15.23:0/372552282 conn(0x7f5ce00f63c0 msgr2=0x7f5ce00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 -- 172.21.15.23:0/372552282 shutdown_connections 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.346+0000 7f5ce581d700 1 -- 172.21.15.23:0/372552282 wait complete. 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.347+0000 7f5ce581d700 1 Processor -- start 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.347+0000 7f5ce581d700 1 -- start start 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.347+0000 7f5ce581d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5ce581d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce00fa820 con 0x7f5ce00703f0 2023-12-08T10:05:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cdeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cdeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41596/0 (socket says 172.21.15.23:41596) 2023-12-08T10:05:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cdeffd700 1 -- 172.21.15.23:0/686120077 learned_addr learned my addr 172.21.15.23:0/686120077 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cdeffd700 1 -- 172.21.15.23:0/686120077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd000b200 con 0x7f5ce00703f0 2023-12-08T10:05:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cdeffd700 1 --2- 172.21.15.23:0/686120077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fc330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5cd000e040 tx=0x7f5cd0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd000d9f0 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce00faa20 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5cd001a9f0 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cd001b8c0 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.348+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce00fae30 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.349+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5cd000f070 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.349+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5cd0025b30 con 0x7f5ce00703f0 2023-12-08T10:05:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.349+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce0045ad0 con 0x7f5ce00703f0 2023-12-08T10:05:37.353 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.351+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5cd0016070 con 0x7f5ce00703f0 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.421+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5ce00ffbd0 con 0x7f5ce00703f0 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.422+0000 7f5cc77fe700 1 -- 172.21.15.23:0/686120077 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f5cd001bdc0 con 0x7f5ce00703f0 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "40b80ed0-95b1-11ee-95a3-87774f69a715", 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.425 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.431 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-08T10:05:37.432 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": false, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:37.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2023-12-08T10:05:29.507833+0000", 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 msgr2=0x7f5ce00fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 --2- 172.21.15.23:0/686120077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fc330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f5cd000e040 tx=0x7f5cd0006ee0).stop 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 shutdown_connections 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 --2- 172.21.15.23:0/686120077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ce00703f0 0x7f5ce00fc330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 >> 172.21.15.23:0/686120077 conn(0x7f5ce00f63c0 msgr2=0x7f5ce00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 shutdown_connections 2023-12-08T10:05:37.434 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:37.423+0000 7f5ce581d700 1 -- 172.21.15.23:0/686120077 wait complete. 2023-12-08T10:05:37.724 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:37.423440+0000 mon.smithi023 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.23:0/686120077' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-08T10:05:37.806 INFO:teuthology.orchestra.run.smithi023.stderr:mgr not available, waiting (2/15)... 2023-12-08T10:05:39.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:39.330045+0000 mon.smithi023 (mon.0) 9 : cluster [INF] Activating manager daemon smithi023.tgiigg 2023-12-08T10:05:39.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:39.337378+0000 mon.smithi023 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi023.tgiigg(active, starting, since 0.00744818s) 2023-12-08T10:05:39.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.338378+0000 mon.smithi023 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.338520+0000 mon.smithi023 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.338632+0000 mon.smithi023 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.338759+0000 mon.smithi023 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.338998+0000 mon.smithi023 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mgr metadata", "who": "smithi023.tgiigg", "id": "smithi023.tgiigg"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:39.352872+0000 mon.smithi023 (mon.0) 16 : cluster [INF] Manager daemon smithi023.tgiigg is now available 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.365294+0000 mon.smithi023 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.366050+0000 mon.smithi023 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.367228+0000 mon.smithi023 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:05:39.683 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.376696+0000 mon.smithi023 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:39.808 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-08T10:05:40.169 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92dac5b700 1 Processor -- start 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92dac5b700 1 -- start start 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92dac5b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92dac5b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d40fee10 con 0x7f92d406fbd0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92d89f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92d89f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43382/0 (socket says 172.21.15.23:43382) 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92d89f7700 1 -- 172.21.15.23:0/3864934068 learned_addr learned my addr 172.21.15.23:0/3864934068 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92d89f7700 1 -- 172.21.15.23:0/3864934068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d40fef50 con 0x7f92d406fbd0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.165+0000 7f92d89f7700 1 --2- 172.21.15.23:0/3864934068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fe920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f92c800b5f0 tx=0x7f92c8006ee0).ready entity=mon.0 client_cookie=cc754d71208c97e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92d37fe700 1 -- 172.21.15.23:0/3864934068 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92c800d600 con 0x7f92d406fbd0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92d37fe700 1 -- 172.21.15.23:0/3864934068 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f92c800dbd0 con 0x7f92d406fbd0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92d37fe700 1 -- 172.21.15.23:0/3864934068 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92c8011c00 con 0x7f92d406fbd0 2023-12-08T10:05:40.170 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 -- 172.21.15.23:0/3864934068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 msgr2=0x7f92d40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 --2- 172.21.15.23:0/3864934068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fe920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f92c800b5f0 tx=0x7f92c8006ee0).stop 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 -- 172.21.15.23:0/3864934068 shutdown_connections 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 --2- 172.21.15.23:0/3864934068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fe920 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 -- 172.21.15.23:0/3864934068 >> 172.21.15.23:0/3864934068 conn(0x7f92d40f63a0 msgr2=0x7f92d40f87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 -- 172.21.15.23:0/3864934068 shutdown_connections 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.166+0000 7f92dac5b700 1 -- 172.21.15.23:0/3864934068 wait complete. 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92dac5b700 1 Processor -- start 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92dac5b700 1 -- start start 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92dac5b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92dac5b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d40fa820 con 0x7f92d406fbd0 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92d89f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92d89f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43398/0 (socket says 172.21.15.23:43398) 2023-12-08T10:05:40.171 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.167+0000 7f92d89f7700 1 -- 172.21.15.23:0/3487546324 learned_addr learned my addr 172.21.15.23:0/3487546324 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92d89f7700 1 -- 172.21.15.23:0/3487546324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c800b200 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92d89f7700 1 --2- 172.21.15.23:0/3487546324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fc3a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f92c800b190 tx=0x7f92c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92c801b410 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d40fa9c0 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f92c801b9e0 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92c801a880 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.168+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d40fadd0 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.169+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f92c800f070 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.169+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f92c8041040 con 0x7f92d406fbd0 2023-12-08T10:05:40.173 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.169+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d4059100 con 0x7f92d406fbd0 2023-12-08T10:05:40.174 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.171+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f92c8025460 con 0x7f92d406fbd0 2023-12-08T10:05:40.248 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.240+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f92d4045ad0 con 0x7f92d406fbd0 2023-12-08T10:05:40.248 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.241+0000 7f92d17fa700 1 -- 172.21.15.23:0/3487546324 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f92c8023e90 con 0x7f92d406fbd0 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "40b80ed0-95b1-11ee-95a3-87774f69a715", 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2023-12-08T10:05:40.249 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-12-08T10:05:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2023-12-08T10:05:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:40.250 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-08T10:05:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-08T10:05:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2023-12-08T10:05:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-08T10:05:40.251 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-08T10:05:40.252 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": false, 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.253 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2023-12-08T10:05:29.507833+0000", 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.241+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 msgr2=0x7f92d40fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.241+0000 7f92dac5b700 1 --2- 172.21.15.23:0/3487546324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fc3a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f92c800b190 tx=0x7f92c8006ee0).stop 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.242+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 shutdown_connections 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.242+0000 7f92dac5b700 1 --2- 172.21.15.23:0/3487546324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92d406fbd0 0x7f92d40fc3a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.242+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 >> 172.21.15.23:0/3487546324 conn(0x7f92d40f63a0 msgr2=0x7f92d40f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.242+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 shutdown_connections 2023-12-08T10:05:40.254 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:40.242+0000 7f92dac5b700 1 -- 172.21.15.23:0/3487546324 wait complete. 2023-12-08T10:05:40.523 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:39.381522+0000 mon.smithi023 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.23:0/1487729482' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:40.524 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:40.242203+0000 mon.smithi023 (mon.0) 22 : audit [DBG] from='client.? 172.21.15.23:0/3487546324' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-08T10:05:40.524 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:40.339883+0000 mon.smithi023 (mon.0) 23 : cluster [DBG] mgrmap e3: smithi023.tgiigg(active, since 1.00995s) 2023-12-08T10:05:40.580 INFO:teuthology.orchestra.run.smithi023.stderr:mgr not available, waiting (3/15)... 2023-12-08T10:05:42.581 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-08T10:05:42.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:42 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:41.386442+0000 mon.smithi023 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi023.tgiigg(active, since 2s) 2023-12-08T10:05:43.090 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.087+0000 7fd8b0db5700 1 Processor -- start 2023-12-08T10:05:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8b0db5700 1 -- start start 2023-12-08T10:05:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8b0db5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8b0db5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0fb490 con 0x7fd8ac0fab80 2023-12-08T10:05:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8aa59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8aa59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43400/0 (socket says 172.21.15.23:43400) 2023-12-08T10:05:43.091 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8aa59c700 1 -- 172.21.15.23:0/1372425396 learned_addr learned my addr 172.21.15.23:0/1372425396 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.088+0000 7fd8aa59c700 1 -- 172.21.15.23:0/1372425396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ac0fb5d0 con 0x7fd8ac0fab80 2023-12-08T10:05:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8aa59c700 1 --2- 172.21.15.23:0/1372425396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac0fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd89c00b5f0 tx=0x7fd89c006ee0).ready entity=mon.0 client_cookie=1d56b1a857f8f3f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:43.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8a959a700 1 -- 172.21.15.23:0/1372425396 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd89c00d600 con 0x7fd8ac0fab80 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8a959a700 1 -- 172.21.15.23:0/1372425396 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd89c00dbd0 con 0x7fd8ac0fab80 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8a959a700 1 -- 172.21.15.23:0/1372425396 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd89c011c00 con 0x7fd8ac0fab80 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 -- 172.21.15.23:0/1372425396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 msgr2=0x7fd8ac0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 --2- 172.21.15.23:0/1372425396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac0fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd89c00b5f0 tx=0x7fd89c006ee0).stop 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 -- 172.21.15.23:0/1372425396 shutdown_connections 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 --2- 172.21.15.23:0/1372425396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac0fafa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 -- 172.21.15.23:0/1372425396 >> 172.21.15.23:0/1372425396 conn(0x7fd8ac0f63c0 msgr2=0x7fd8ac0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 -- 172.21.15.23:0/1372425396 shutdown_connections 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.089+0000 7fd8b0db5700 1 -- 172.21.15.23:0/1372425396 wait complete. 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8b0db5700 1 Processor -- start 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8b0db5700 1 -- start start 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8b0db5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8b0db5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ac0ff290 con 0x7fd8ac0fab80 2023-12-08T10:05:43.095 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8aa59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8aa59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43416/0 (socket says 172.21.15.23:43416) 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8aa59c700 1 -- 172.21.15.23:0/427663288 learned_addr learned my addr 172.21.15.23:0/427663288 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.090+0000 7fd8aa59c700 1 -- 172.21.15.23:0/427663288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c00b200 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.091+0000 7fd8aa59c700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac100e10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd89c00e040 tx=0x7fd89c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.091+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd89c01b410 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.091+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd89c01b9e0 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.091+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd89c01a820 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.091+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ac0ff430 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.091+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ac0ff840 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.092+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fd89c00f070 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.092+0000 7fd8a2ffd700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7fd88c02fd50 0x7fd88c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.092+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd89c0412c0 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.092+0000 7fd8a9d9b700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7fd88c02fd50 0x7fd88c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.092+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ac059100 con 0x7fd8ac0fab80 2023-12-08T10:05:43.096 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.092+0000 7fd8a9d9b700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7fd88c02fd50 0x7fd88c032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd8940096a0 tx=0x7fd894006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:43.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.097+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd89c016020 con 0x7fd8ac0fab80 2023-12-08T10:05:43.321 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.312+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd8ac045ad0 con 0x7fd8ac0fab80 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.313+0000 7fd8a2ffd700 1 -- 172.21.15.23:0/427663288 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd89c0164e0 con 0x7fd8ac0fab80 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsid": "40b80ed0-95b1-11ee-95a3-87774f69a715", 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "health": { 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "checks": {}, 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mutes": [] 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.322 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum": [ 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 0 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "smithi023" 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "monmap": { 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-08T10:05:43.323 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-08T10:05:43.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osdmap": { 2023-12-08T10:05:43.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:43.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-08T10:05:43.325 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgmap": { 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-08T10:05:43.326 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "fsmap": { 2023-12-08T10:05:43.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:43.327 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": true, 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modules": [ 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "iostat", 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "restful" 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ], 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "servicemap": { 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-08T10:05:43.328 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "modified": "2023-12-08T10:05:29.507833+0000", 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "services": {} 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: }, 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7fd88c02fd50 msgr2=0x7fd88c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7fd88c02fd50 0x7fd88c032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd8940096a0 tx=0x7fd894006b40).stop 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 msgr2=0x7fd8ac100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac100e10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd89c00e040 tx=0x7fd89c006ee0).stop 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 shutdown_connections 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7fd88c02fd50 0x7fd88c032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 --2- 172.21.15.23:0/427663288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8ac0fab80 0x7fd8ac100e10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 >> 172.21.15.23:0/427663288 conn(0x7fd8ac0f63c0 msgr2=0x7fd8ac070010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 shutdown_connections 2023-12-08T10:05:43.329 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:43.315+0000 7fd8b0db5700 1 -- 172.21.15.23:0/427663288 wait complete. 2023-12-08T10:05:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:mgr is available 2023-12-08T10:05:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:Enabling cephadm module... 2023-12-08T10:05:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-12-08T10:05:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.015+0000 7faa1e4ff700 1 Processor -- start 2023-12-08T10:05:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.015+0000 7faa1e4ff700 1 -- start start 2023-12-08T10:05:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa1e4ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa1e4ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180fedd0 con 0x7faa18070430 2023-12-08T10:05:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa17fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa17fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43432/0 (socket says 172.21.15.23:43432) 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa17fff700 1 -- 172.21.15.23:0/3179195153 learned_addr learned my addr 172.21.15.23:0/3179195153 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa17fff700 1 -- 172.21.15.23:0/3179195153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180fef10 con 0x7faa18070430 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa17fff700 1 --2- 172.21.15.23:0/3179195153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fe8e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7faa0800b5f0 tx=0x7faa08006ee0).ready entity=mon.0 client_cookie=4efa8ae1897ee14b server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa16ffd700 1 -- 172.21.15.23:0/3179195153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0800d600 con 0x7faa18070430 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa16ffd700 1 -- 172.21.15.23:0/3179195153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faa0800dbd0 con 0x7faa18070430 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.016+0000 7faa16ffd700 1 -- 172.21.15.23:0/3179195153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa08011c00 con 0x7faa18070430 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 -- 172.21.15.23:0/3179195153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 msgr2=0x7faa180fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 --2- 172.21.15.23:0/3179195153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fe8e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7faa0800b5f0 tx=0x7faa08006ee0).stop 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 -- 172.21.15.23:0/3179195153 shutdown_connections 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 --2- 172.21.15.23:0/3179195153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fe8e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 -- 172.21.15.23:0/3179195153 >> 172.21.15.23:0/3179195153 conn(0x7faa180f63c0 msgr2=0x7faa180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 -- 172.21.15.23:0/3179195153 shutdown_connections 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 -- 172.21.15.23:0/3179195153 wait complete. 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.017+0000 7faa1e4ff700 1 Processor -- start 2023-12-08T10:05:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa1e4ff700 1 -- start start 2023-12-08T10:05:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa1e4ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa1e4ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa180fa7e0 con 0x7faa18070430 2023-12-08T10:05:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa17fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa17fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43448/0 (socket says 172.21.15.23:43448) 2023-12-08T10:05:44.022 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa17fff700 1 -- 172.21.15.23:0/2845369547 learned_addr learned my addr 172.21.15.23:0/2845369547 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa17fff700 1 -- 172.21.15.23:0/2845369547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0800b200 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.018+0000 7faa17fff700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fc360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7faa0800b190 tx=0x7faa08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.019+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0801b410 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.019+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faa0801b9e0 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.019+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa180fa980 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.019+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa0801a880 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.019+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa180fad90 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.019+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7faa0800f070 con 0x7faa18070430 2023-12-08T10:05:44.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.020+0000 7faa14ff9700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7faa0402fd50 0x7faa04032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:44.024 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.020+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7faa08041ff0 con 0x7faa18070430 2023-12-08T10:05:44.024 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.020+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa18045ad0 con 0x7faa18070430 2023-12-08T10:05:44.024 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.020+0000 7faa177fe700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7faa0402fd50 0x7faa04032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:44.024 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.020+0000 7faa177fe700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7faa0402fd50 0x7faa04032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faa0c0096a0 tx=0x7faa0c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:44.026 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.024+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa08016070 con 0x7faa18070430 2023-12-08T10:05:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:43.314287+0000 mon.smithi023 (mon.0) 25 : audit [DBG] from='client.? 172.21.15.23:0/427663288' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-08T10:05:44.222 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.220+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7faa18027a00 con 0x7faa18070430 2023-12-08T10:05:44.793 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.790+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7faa080257e0 con 0x7faa18070430 2023-12-08T10:05:44.794 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.792+0000 7faa14ff9700 1 -- 172.21.15.23:0/2845369547 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7faa08006a70 con 0x7faa18070430 2023-12-08T10:05:44.796 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7faa0402fd50 msgr2=0x7faa04032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:44.796 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7faa0402fd50 0x7faa04032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faa0c0096a0 tx=0x7faa0c006b40).stop 2023-12-08T10:05:44.796 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 msgr2=0x7faa180fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:44.796 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fc360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7faa0800b190 tx=0x7faa08006ee0).stop 2023-12-08T10:05:44.797 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 shutdown_connections 2023-12-08T10:05:44.797 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7faa0402fd50 0x7faa04032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:44.797 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 --2- 172.21.15.23:0/2845369547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa18070430 0x7faa180fc360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:44.797 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 >> 172.21.15.23:0/2845369547 conn(0x7faa180f63c0 msgr2=0x7faa180f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:44.797 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 shutdown_connections 2023-12-08T10:05:44.797 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:44.794+0000 7faa1e4ff700 1 -- 172.21.15.23:0/2845369547 wait complete. 2023-12-08T10:05:45.064 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:44 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:44.221844+0000 mon.smithi023 (mon.0) 26 : audit [INF] from='client.? 172.21.15.23:0/2845369547' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-08T10:05:45.129 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-08T10:05:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f875cc62700 1 Processor -- start 2023-12-08T10:05:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f875cc62700 1 -- start start 2023-12-08T10:05:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f875cc62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f87580fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f875cc62700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580fec20 con 0x7f8758070430 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f87577fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f87580fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f87577fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f87580fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43470/0 (socket says 172.21.15.23:43470) 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.473+0000 7f87577fe700 1 -- 172.21.15.23:0/3642066025 learned_addr learned my addr 172.21.15.23:0/3642066025 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.474+0000 7f87577fe700 1 -- 172.21.15.23:0/3642066025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87580fed60 con 0x7f8758070430 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.474+0000 7f87577fe700 1 --2- 172.21.15.23:0/3642066025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f87580fe730 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f874800b5f0 tx=0x7f8748006ee0).ready entity=mon.0 client_cookie=608eedd90d087704 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.474+0000 7f87567fc700 1 -- 172.21.15.23:0/3642066025 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f874800d600 con 0x7f8758070430 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.474+0000 7f87567fc700 1 -- 172.21.15.23:0/3642066025 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f874800dbd0 con 0x7f8758070430 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.474+0000 7f87567fc700 1 -- 172.21.15.23:0/3642066025 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8748011c00 con 0x7f8758070430 2023-12-08T10:05:45.478 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 -- 172.21.15.23:0/3642066025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 msgr2=0x7f87580fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 --2- 172.21.15.23:0/3642066025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f87580fe730 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f874800b5f0 tx=0x7f8748006ee0).stop 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 -- 172.21.15.23:0/3642066025 shutdown_connections 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 --2- 172.21.15.23:0/3642066025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f87580fe730 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 -- 172.21.15.23:0/3642066025 >> 172.21.15.23:0/3642066025 conn(0x7f87580f6210 msgr2=0x7f87580f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 -- 172.21.15.23:0/3642066025 shutdown_connections 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.475+0000 7f875cc62700 1 -- 172.21.15.23:0/3642066025 wait complete. 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f875cc62700 1 Processor -- start 2023-12-08T10:05:45.479 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f875cc62700 1 -- start start 2023-12-08T10:05:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f875cc62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f875810a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f875cc62700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875810ad20 con 0x7f8758070430 2023-12-08T10:05:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f87577fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f875810a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:45.481 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f87577fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f875810a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43486/0 (socket says 172.21.15.23:43486) 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.476+0000 7f87577fe700 1 -- 172.21.15.23:0/2118431814 learned_addr learned my addr 172.21.15.23:0/2118431814 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f87577fe700 1 -- 172.21.15.23:0/2118431814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874800b200 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f87577fe700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f875810a830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f874800b190 tx=0x7f8748006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f874801b410 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f874801b9e0 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875810af20 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f874801a880 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.477+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875810b330 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.478+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f874800f070 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.478+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8758045ad0 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.478+0000 7f873ffff700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 0x7f8744032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.478+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f87480412a0 con 0x7f8758070430 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.478+0000 7f8756ffd700 1 -- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 msgr2=0x7f8744032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4281746080 2023-12-08T10:05:45.482 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.478+0000 7f8756ffd700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 0x7f8744032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-08T10:05:45.484 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.482+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8748010360 con 0x7f8758070430 2023-12-08T10:05:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.679+0000 7f8756ffd700 1 -- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 msgr2=0x7f8744032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4281746080 2023-12-08T10:05:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.679+0000 7f8756ffd700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 0x7f8744032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-08T10:05:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.680+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f875810c3e0 con 0x7f8758070430 2023-12-08T10:05:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.681+0000 7f873ffff700 1 -- 172.21.15.23:0/2118431814 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f8748016070 con 0x7f8758070430 2023-12-08T10:05:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:05:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-08T10:05:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": true, 2023-12-08T10:05:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "active_name": "smithi023.tgiigg", 2023-12-08T10:05:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-08T10:05:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 msgr2=0x7f8744032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 0x7f8744032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 msgr2=0x7f875810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f875810a830 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f874800b190 tx=0x7f8748006ee0).stop 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 shutdown_connections 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f874402fd50 0x7f8744032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 --2- 172.21.15.23:0/2118431814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8758070430 0x7f875810a830 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.683+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 >> 172.21.15.23:0/2118431814 conn(0x7f87580f6210 msgr2=0x7f87580f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.684+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 shutdown_connections 2023-12-08T10:05:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:45.684+0000 7f875cc62700 1 -- 172.21.15.23:0/2118431814 wait complete. 2023-12-08T10:05:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:44.793286+0000 mon.smithi023 (mon.0) 27 : audit [INF] from='client.? 172.21.15.23:0/2845369547' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-08T10:05:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:44.793351+0000 mon.smithi023 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi023.tgiigg(active, since 5s) 2023-12-08T10:05:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:45.682377+0000 mon.smithi023 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.23:0/2118431814' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-08T10:05:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for the mgr to restart... 2023-12-08T10:05:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr epoch 5... 2023-12-08T10:05:47.138 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.121+0000 7f7b8fc37700 1 Processor -- start 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.121+0000 7f7b8fc37700 1 -- start start 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8fc37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b880fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8fc37700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880fb490 con 0x7f7b880fab80 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8d9d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b880fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8d9d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b880fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43498/0 (socket says 172.21.15.23:43498) 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8d9d3700 1 -- 172.21.15.23:0/2176007346 learned_addr learned my addr 172.21.15.23:0/2176007346 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8d9d3700 1 -- 172.21.15.23:0/2176007346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b880fb5d0 con 0x7f7b880fab80 2023-12-08T10:05:48.125 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.122+0000 7f7b8d9d3700 1 --2- 172.21.15.23:0/2176007346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b880fafa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7b8400b870 tx=0x7f7b84006f70).ready entity=mon.0 client_cookie=e774e0fe36eec2f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:48.126 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8c9d1700 1 -- 172.21.15.23:0/2176007346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b8400da20 con 0x7f7b880fab80 2023-12-08T10:05:48.126 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8c9d1700 1 -- 172.21.15.23:0/2176007346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7b84010470 con 0x7f7b880fab80 2023-12-08T10:05:48.126 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8c9d1700 1 -- 172.21.15.23:0/2176007346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b8400f5b0 con 0x7f7b880fab80 2023-12-08T10:05:48.128 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8fc37700 1 -- 172.21.15.23:0/2176007346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 msgr2=0x7f7b880fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:48.128 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/2176007346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b880fafa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7b8400b870 tx=0x7f7b84006f70).stop 2023-12-08T10:05:48.128 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8fc37700 1 -- 172.21.15.23:0/2176007346 shutdown_connections 2023-12-08T10:05:48.128 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/2176007346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b880fafa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.123+0000 7f7b8fc37700 1 -- 172.21.15.23:0/2176007346 >> 172.21.15.23:0/2176007346 conn(0x7f7b880f63c0 msgr2=0x7f7b880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.124+0000 7f7b8fc37700 1 -- 172.21.15.23:0/2176007346 shutdown_connections 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.124+0000 7f7b8fc37700 1 -- 172.21.15.23:0/2176007346 wait complete. 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.124+0000 7f7b8fc37700 1 Processor -- start 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.124+0000 7f7b8fc37700 1 -- start start 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8fc37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b8810cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8fc37700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8810d070 con 0x7f7b880fab80 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8d9d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b8810cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8d9d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b8810cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43500/0 (socket says 172.21.15.23:43500) 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8d9d3700 1 -- 172.21.15.23:0/3090248985 learned_addr learned my addr 172.21.15.23:0/3090248985 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8d9d3700 1 -- 172.21.15.23:0/3090248985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8400b480 con 0x7f7b880fab80 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8d9d3700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b8810cb80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7b84006a40 tx=0x7f7b84006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b8400de00 con 0x7f7b880fab80 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8810d270 con 0x7f7b880fab80 2023-12-08T10:05:48.129 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7b8400fef0 con 0x7f7b880fab80 2023-12-08T10:05:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b84019e90 con 0x7f7b880fab80 2023-12-08T10:05:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.125+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8810d680 con 0x7f7b880fab80 2023-12-08T10:05:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.126+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f7b8402a070 con 0x7f7b880fab80 2023-12-08T10:05:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.126+0000 7f7b767fc700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.126+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 --> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b84019e90 con 0x7f7b7002fd50 2023-12-08T10:05:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.127+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7b84041b30 con 0x7f7b880fab80 2023-12-08T10:05:48.131 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.127+0000 7f7b8d1d2700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 msgr2=0x7f7b70032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4281746080 2023-12-08T10:05:48.131 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.127+0000 7f7b8d1d2700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-08T10:05:48.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.328+0000 7f7b8d1d2700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 msgr2=0x7f7b70032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4281746080 2023-12-08T10:05:48.330 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.328+0000 7f7b8d1d2700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-08T10:05:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.729+0000 7f7b8d1d2700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 msgr2=0x7f7b70032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4281746080 2023-12-08T10:05:48.731 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:48.729+0000 7f7b8d1d2700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-08T10:05:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:49.530+0000 7f7b8d1d2700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 msgr2=0x7f7b70032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4281746080 2023-12-08T10:05:49.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:49.530+0000 7f7b8d1d2700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-08T10:05:50.026 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:50.024+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f7b84016030 con 0x7f7b880fab80 2023-12-08T10:05:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:50.024+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 msgr2=0x7f7b70032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-08T10:05:50.027 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:50.024+0000 7f7b767fc700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:05:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:49.973313+0000 mon.smithi023 (mon.0) 30 : cluster [INF] Active manager daemon smithi023.tgiigg restarted 2023-12-08T10:05:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:49.973815+0000 mon.smithi023 (mon.0) 31 : cluster [INF] Activating manager daemon smithi023.tgiigg 2023-12-08T10:05:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:49.977261+0000 mon.smithi023 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-08T10:05:51.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.035+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f7b84019630 con 0x7f7b880fab80 2023-12-08T10:05:51.038 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.035+0000 7f7b767fc700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f7b700336b0 0x7f7b70035aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.035+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b84019e90 con 0x7f7b700336b0 2023-12-08T10:05:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.036+0000 7f7b8d1d2700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f7b700336b0 0x7f7b70035aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.036+0000 7f7b8d1d2700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f7b700336b0 0x7f7b70035aa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7b780036d0 tx=0x7f7b78006ca0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.037+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f7b84019e90 con 0x7f7b700336b0 2023-12-08T10:05:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7b88059100 con 0x7f7b700336b0 2023-12-08T10:05:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b767fc700 1 -- 172.21.15.23:0/3090248985 <== mgr.14116 v2:172.21.15.23:6800/4062254273 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f7b88059100 con 0x7f7b700336b0 2023-12-08T10:05:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:05:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-08T10:05:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "initialized": true 2023-12-08T10:05:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:05:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f7b700336b0 msgr2=0x7f7b70035aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f7b700336b0 0x7f7b70035aa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7b780036d0 tx=0x7f7b78006ca0).stop 2023-12-08T10:05:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 msgr2=0x7f7b8810cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b8810cb80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7b84006a40 tx=0x7f7b84006f70).stop 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 shutdown_connections 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f7b700336b0 0x7f7b70035aa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b880fab80 0x7f7b8810cb80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 --2- 172.21.15.23:0/3090248985 >> [v2:172.21.15.23:6800/4281746080,v1:172.21.15.23:6801/4281746080] conn(0x7f7b7002fd50 0x7f7b70032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.044+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 >> 172.21.15.23:0/3090248985 conn(0x7f7b880f63c0 msgr2=0x7f7b880ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.045+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 shutdown_connections 2023-12-08T10:05:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.045+0000 7f7b8fc37700 1 -- 172.21.15.23:0/3090248985 wait complete. 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:50.026472+0000 mon.smithi023 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi023.tgiigg(active, starting, since 0.0527332s) 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.029032+0000 mon.smithi023 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.029236+0000 mon.smithi023 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mgr metadata", "who": "smithi023.tgiigg", "id": "smithi023.tgiigg"}]: dispatch 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.029413+0000 mon.smithi023 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.029558+0000 mon.smithi023 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.029684+0000 mon.smithi023 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:50.043827+0000 mon.smithi023 (mon.0) 39 : cluster [INF] Manager daemon smithi023.tgiigg is now available 2023-12-08T10:05:51.323 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.057445+0000 mon.smithi023 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.060332+0000 mon.smithi023 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.063031+0000 mon.smithi023 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.160330+0000 mon.smithi023 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.161053+0000 mon.smithi023 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.162925+0000 mon.smithi023 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.164426+0000 mon.smithi023 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.174978+0000 mon.smithi023 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:05:51.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:50.176281+0000 mon.smithi023 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:05:51.370 INFO:teuthology.orchestra.run.smithi023.stderr:mgr epoch 5 is available 2023-12-08T10:05:51.370 INFO:teuthology.orchestra.run.smithi023.stderr:Setting orchestrator backend to cephadm... 2023-12-08T10:05:51.371 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-12-08T10:05:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.711+0000 7f8476632700 1 Processor -- start 2023-12-08T10:05:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.711+0000 7f8476632700 1 -- start start 2023-12-08T10:05:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f8476632700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f8470100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f8476632700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84700fc770 con 0x7f8470070390 2023-12-08T10:05:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f8470100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:51.715 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f8470100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57806/0 (socket says 172.21.15.23:57806) 2023-12-08T10:05:51.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846ffff700 1 -- 172.21.15.23:0/3005250741 learned_addr learned my addr 172.21.15.23:0/3005250741 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:51.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846ffff700 1 -- 172.21.15.23:0/3005250741 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8470101270 con 0x7f8470070390 2023-12-08T10:05:51.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846ffff700 1 --2- 172.21.15.23:0/3005250741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f8470100d80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f846000b5f0 tx=0x7f8460006ee0).ready entity=mon.0 client_cookie=262fbb7ef6beda2e server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:51.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846effd700 1 -- 172.21.15.23:0/3005250741 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846000d600 con 0x7f8470070390 2023-12-08T10:05:51.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846effd700 1 -- 172.21.15.23:0/3005250741 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f846000dbd0 con 0x7f8470070390 2023-12-08T10:05:51.716 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.712+0000 7f846effd700 1 -- 172.21.15.23:0/3005250741 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8460011c00 con 0x7f8470070390 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 -- 172.21.15.23:0/3005250741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 msgr2=0x7f8470100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 --2- 172.21.15.23:0/3005250741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f8470100d80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f846000b5f0 tx=0x7f8460006ee0).stop 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 -- 172.21.15.23:0/3005250741 shutdown_connections 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 --2- 172.21.15.23:0/3005250741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f8470100d80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 -- 172.21.15.23:0/3005250741 >> 172.21.15.23:0/3005250741 conn(0x7f84700f63c0 msgr2=0x7f84700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 -- 172.21.15.23:0/3005250741 shutdown_connections 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.713+0000 7f8476632700 1 -- 172.21.15.23:0/3005250741 wait complete. 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f8476632700 1 Processor -- start 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f8476632700 1 -- start start 2023-12-08T10:05:51.719 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f8476632700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f847010cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f8476632700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847010d0f0 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f846ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f847010cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f846ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f847010cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57818/0 (socket says 172.21.15.23:57818) 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.714+0000 7f846ffff700 1 -- 172.21.15.23:0/2498278091 learned_addr learned my addr 172.21.15.23:0/2498278091 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f846ffff700 1 -- 172.21.15.23:0/2498278091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846000b200 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f846ffff700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f847010cc00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f846000b190 tx=0x7f8460006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846001b410 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847010d2f0 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f846001b9e0 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f846001a880 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.715+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847010d700 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.716+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8470045ad0 con 0x7f8470070390 2023-12-08T10:05:51.720 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.716+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f846000f070 con 0x7f8470070390 2023-12-08T10:05:51.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.717+0000 7f846cff9700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f845002fca0 0x7f8450032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:51.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.717+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8460041c00 con 0x7f8470070390 2023-12-08T10:05:51.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.717+0000 7f846f7fe700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f845002fca0 0x7f8450032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:51.721 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.717+0000 7f846f7fe700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f845002fca0 0x7f8450032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f84580096a0 tx=0x7f8458006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:51.722 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.721+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f846002db70 con 0x7f8470070390 2023-12-08T10:05:51.911 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.909+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f847010df70 con 0x7f845002fca0 2023-12-08T10:05:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.916+0000 7f846cff9700 1 -- 172.21.15.23:0/2498278091 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f847010df70 con 0x7f845002fca0 2023-12-08T10:05:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f845002fca0 msgr2=0x7f8450032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f845002fca0 0x7f8450032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f84580096a0 tx=0x7f8458006b40).stop 2023-12-08T10:05:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 msgr2=0x7f847010cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f847010cc00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f846000b190 tx=0x7f8460006ee0).stop 2023-12-08T10:05:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 shutdown_connections 2023-12-08T10:05:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f845002fca0 0x7f8450032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 --2- 172.21.15.23:0/2498278091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8470070390 0x7f847010cc00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.918+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 >> 172.21.15.23:0/2498278091 conn(0x7f84700f63c0 msgr2=0x7f84700f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.919+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 shutdown_connections 2023-12-08T10:05:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:51.919+0000 7f8476632700 1 -- 172.21.15.23:0/2498278091 wait complete. 2023-12-08T10:05:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-12-08T10:05:52.366 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:51.038146+0000 mgr.smithi023.tgiigg (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-08T10:05:52.366 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:51.038898+0000 mon.smithi023 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi023.tgiigg(active, since 1.06515s) 2023-12-08T10:05:52.366 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:51.045480+0000 mgr.smithi023.tgiigg (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-08T10:05:52.367 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:51.917621+0000 mon.smithi023 (mon.0) 50 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:52.367 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:51.920935+0000 mon.smithi023 (mon.0) 51 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd181f23700 1 Processor -- start 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd181f23700 1 -- start start 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd181f23700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd181f23700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0fc750 con 0x7fd17c0703b0 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd17b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd17b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57822/0 (socket says 172.21.15.23:57822) 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd17b7fe700 1 -- 172.21.15.23:0/2766981780 learned_addr learned my addr 172.21.15.23:0/2766981780 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd17b7fe700 1 -- 172.21.15.23:0/2766981780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c101250 con 0x7fd17c0703b0 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.609+0000 7fd17b7fe700 1 --2- 172.21.15.23:0/2766981780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd17400b5f0 tx=0x7fd174006ee0).ready entity=mon.0 client_cookie=e888022ed0722d98 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd17a7fc700 1 -- 172.21.15.23:0/2766981780 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17400d600 con 0x7fd17c0703b0 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd17a7fc700 1 -- 172.21.15.23:0/2766981780 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd17400dbd0 con 0x7fd17c0703b0 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd17a7fc700 1 -- 172.21.15.23:0/2766981780 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd174011c00 con 0x7fd17c0703b0 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 -- 172.21.15.23:0/2766981780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 msgr2=0x7fd17c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 --2- 172.21.15.23:0/2766981780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c100d60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fd17400b5f0 tx=0x7fd174006ee0).stop 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 -- 172.21.15.23:0/2766981780 shutdown_connections 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 --2- 172.21.15.23:0/2766981780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c100d60 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 -- 172.21.15.23:0/2766981780 >> 172.21.15.23:0/2766981780 conn(0x7fd17c0f63c0 msgr2=0x7fd17c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 -- 172.21.15.23:0/2766981780 shutdown_connections 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.610+0000 7fd181f23700 1 -- 172.21.15.23:0/2766981780 wait complete. 2023-12-08T10:05:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.611+0000 7fd181f23700 1 Processor -- start 2023-12-08T10:05:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.611+0000 7fd181f23700 1 -- start start 2023-12-08T10:05:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.611+0000 7fd181f23700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.611+0000 7fd181f23700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c10d100 con 0x7fd17c0703b0 2023-12-08T10:05:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd17b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd17b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57838/0 (socket says 172.21.15.23:57838) 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd17b7fe700 1 -- 172.21.15.23:0/156826442 learned_addr learned my addr 172.21.15.23:0/156826442 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd17b7fe700 1 -- 172.21.15.23:0/156826442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17400b200 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd17b7fe700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c10cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd17400b190 tx=0x7fd174006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17401b410 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd17401b9e0 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c10d300 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17401a880 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.612+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c10d710 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.613+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fd17400f070 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.613+0000 7fd173fff700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7fd15c02fca0 0x7fd15c032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.613+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd1740411b0 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.613+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c045ad0 con 0x7fd17c0703b0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.613+0000 7fd17affd700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7fd15c02fca0 0x7fd15c032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.614+0000 7fd17affd700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7fd15c02fca0 0x7fd15c032160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd1640096a0 tx=0x7fd164006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.618+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd174016070 con 0x7fd17c0703b0 2023-12-08T10:05:52.778 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.776+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd17c106b70 con 0x7fd15c02fca0 2023-12-08T10:05:52.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.777+0000 7fd173fff700 1 -- 172.21.15.23:0/156826442 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd17c106b70 con 0x7fd15c02fca0 2023-12-08T10:05:52.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: value unchanged 2023-12-08T10:05:52.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7fd15c02fca0 msgr2=0x7fd15c032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:52.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7fd15c02fca0 0x7fd15c032160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd1640096a0 tx=0x7fd164006b40).stop 2023-12-08T10:05:52.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 msgr2=0x7fd17c10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:52.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c10cc10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd17400b190 tx=0x7fd174006ee0).stop 2023-12-08T10:05:52.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 shutdown_connections 2023-12-08T10:05:52.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7fd15c02fca0 0x7fd15c032160 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:52.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 --2- 172.21.15.23:0/156826442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd17c0703b0 0x7fd17c10cc10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:52.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.778+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 >> 172.21.15.23:0/156826442 conn(0x7fd17c0f63c0 msgr2=0x7fd17c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:52.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.779+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 shutdown_connections 2023-12-08T10:05:52.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:52.779+0000 7fd181f23700 1 -- 172.21.15.23:0/156826442 wait complete. 2023-12-08T10:05:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:Generating ssh key... 2023-12-08T10:05:53.112 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8bfff700 1 Processor -- start 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8bfff700 1 -- start start 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c0fb2e0 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57850/0 (socket says 172.21.15.23:57850) 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.766+0000 7f1b8affd700 1 -- 172.21.15.23:0/2635765205 learned_addr learned my addr 172.21.15.23:0/2635765205 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:53.789 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8affd700 1 -- 172.21.15.23:0/2635765205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c0fb420 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8affd700 1 --2- 172.21.15.23:0/2635765205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c0fadf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f1b7c00b870 tx=0x7f1b7c006f70).ready entity=mon.0 client_cookie=ead93f9a8e5c3531 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b89ffb700 1 -- 172.21.15.23:0/2635765205 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c00da20 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b89ffb700 1 -- 172.21.15.23:0/2635765205 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b7c010470 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b89ffb700 1 -- 172.21.15.23:0/2635765205 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c00f5b0 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 -- 172.21.15.23:0/2635765205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 msgr2=0x7f1b8c0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 --2- 172.21.15.23:0/2635765205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c0fadf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f1b7c00b870 tx=0x7f1b7c006f70).stop 2023-12-08T10:05:53.790 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 -- 172.21.15.23:0/2635765205 shutdown_connections 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 --2- 172.21.15.23:0/2635765205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c0fadf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 -- 172.21.15.23:0/2635765205 >> 172.21.15.23:0/2635765205 conn(0x7f1b8c0f6210 msgr2=0x7f1b8c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 -- 172.21.15.23:0/2635765205 shutdown_connections 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.767+0000 7f1b8bfff700 1 -- 172.21.15.23:0/2635765205 wait complete. 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8bfff700 1 Processor -- start 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8bfff700 1 -- start start 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c108570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c108a60 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c108570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c108570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57858/0 (socket says 172.21.15.23:57858) 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.768+0000 7f1b8affd700 1 -- 172.21.15.23:0/3576087595 learned_addr learned my addr 172.21.15.23:0/3576087595 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b8affd700 1 -- 172.21.15.23:0/3576087595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c00b480 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b8affd700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c108570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f1b7c006a40 tx=0x7f1b7c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c00f3d0 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1b7c00fef0 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c108c60 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b7c019d40 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.769+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c109070 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.770+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1b7c02b060 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.770+0000 7f1b737fe700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1b7802fdb0 0x7f1b78032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.770+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1b7c0440f0 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.770+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c045ad0 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.770+0000 7f1b8a7fc700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1b7802fdb0 0x7f1b78032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:53.792 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.770+0000 7f1b8a7fc700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1b7802fdb0 0x7f1b78032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1b800096a0 tx=0x7f1b80006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:53.793 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.774+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b7c014080 con 0x7f1b8c0fa9d0 2023-12-08T10:05:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:53.938+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b8c102490 con 0x7f1b7802fdb0 2023-12-08T10:05:54.137 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:51.911647+0000 mgr.smithi023.tgiigg (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:05:54.137 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:52.778209+0000 mgr.smithi023.tgiigg (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:05:54.137 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:05:52.919949+0000 mon.smithi023 (mon.0) 52 : cluster [DBG] mgrmap e8: smithi023.tgiigg(active, since 2s) 2023-12-08T10:05:54.147 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.145+0000 7f1b737fe700 1 -- 172.21.15.23:0/3576087595 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1b8c102490 con 0x7f1b7802fdb0 2023-12-08T10:05:54.149 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1b7802fdb0 msgr2=0x7f1b78032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:54.149 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1b7802fdb0 0x7f1b78032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f1b800096a0 tx=0x7f1b80006b40).stop 2023-12-08T10:05:54.149 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 msgr2=0x7f1b8c108570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:54.149 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c108570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f1b7c006a40 tx=0x7f1b7c006f70).stop 2023-12-08T10:05:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 shutdown_connections 2023-12-08T10:05:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1b7802fdb0 0x7f1b78032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 --2- 172.21.15.23:0/3576087595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b8c0fa9d0 0x7f1b8c108570 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.146+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 >> 172.21.15.23:0/3576087595 conn(0x7f1b8c0f6210 msgr2=0x7f1b8c0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.147+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 shutdown_connections 2023-12-08T10:05:54.150 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:54.147+0000 7f1b8bfff700 1 -- 172.21.15.23:0/3576087595 wait complete. 2023-12-08T10:05:55.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:05:53.939961+0000 mgr.smithi023.tgiigg (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-12-08T10:05:55.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:54.141470+0000 mon.smithi023 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:55.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:54.145473+0000 mon.smithi023 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:05:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-12-08T10:05:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.907+0000 7f1bfa1b7700 1 Processor -- start 2023-12-08T10:05:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.907+0000 7f1bfa1b7700 1 -- start start 2023-12-08T10:05:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bfa1b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf4100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bfa1b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40fc710 con 0x7f1bf40703f0 2023-12-08T10:05:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf4100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:58.004 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf4100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57866/0 (socket says 172.21.15.23:57866) 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf37fe700 1 -- 172.21.15.23:0/1405677376 learned_addr learned my addr 172.21.15.23:0/1405677376 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf37fe700 1 -- 172.21.15.23:0/1405677376 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4101210 con 0x7f1bf40703f0 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf37fe700 1 --2- 172.21.15.23:0/1405677376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf4100d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1bdc00b5f0 tx=0x7f1bdc006ee0).ready entity=mon.0 client_cookie=e25e870ace7122de server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf27fc700 1 -- 172.21.15.23:0/1405677376 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bdc00d600 con 0x7f1bf40703f0 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf27fc700 1 -- 172.21.15.23:0/1405677376 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1bdc00dbd0 con 0x7f1bf40703f0 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.908+0000 7f1bf27fc700 1 -- 172.21.15.23:0/1405677376 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bdc011c00 con 0x7f1bf40703f0 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1405677376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 msgr2=0x7f1bf4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 --2- 172.21.15.23:0/1405677376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf4100d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1bdc00b5f0 tx=0x7f1bdc006ee0).stop 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1405677376 shutdown_connections 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 --2- 172.21.15.23:0/1405677376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf4100d20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1405677376 >> 172.21.15.23:0/1405677376 conn(0x7f1bf40f63c0 msgr2=0x7f1bf40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1405677376 shutdown_connections 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.909+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1405677376 wait complete. 2023-12-08T10:05:58.005 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bfa1b7700 1 Processor -- start 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bfa1b7700 1 -- start start 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bfa1b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf410cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bfa1b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf410d0b0 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bf37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf410cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bf37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf410cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57868/0 (socket says 172.21.15.23:57868) 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bf37fe700 1 -- 172.21.15.23:0/1375400730 learned_addr learned my addr 172.21.15.23:0/1375400730 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.910+0000 7f1bf37fe700 1 -- 172.21.15.23:0/1375400730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc00b200 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.911+0000 7f1bf37fe700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf410cbc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1bdc00b190 tx=0x7f1bdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.911+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bdc01b410 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.911+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1bdc01b9e0 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.911+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf410d2b0 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.911+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bdc01a880 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.911+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf410d6c0 con 0x7f1bf40703f0 2023-12-08T10:05:58.006 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.912+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf4045ad0 con 0x7f1bf40703f0 2023-12-08T10:05:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.912+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1bdc00f070 con 0x7f1bf40703f0 2023-12-08T10:05:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.912+0000 7f1bebfff700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1bd402fd50 0x7f1bd4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:05:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.912+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1bdc041cf0 con 0x7f1bf40703f0 2023-12-08T10:05:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.912+0000 7f1bf2ffd700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1bd402fd50 0x7f1bd4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:05:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.913+0000 7f1bf2ffd700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1bd402fd50 0x7f1bd4032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1be40096a0 tx=0x7f1be4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:05:58.007 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:57.916+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bdc02db70 con 0x7f1bf40703f0 2023-12-08T10:05:58.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.083+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bf410dfa0 con 0x7f1bd402fd50 2023-12-08T10:05:58.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.084+0000 7f1bebfff700 1 -- 172.21.15.23:0/1375400730 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f1bf410dfa0 con 0x7f1bd402fd50 2023-12-08T10:05:58.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4bDrwPrmqPrYrkAh/poL1VSh81f4XD9qxCy/Zfz0xIbScLW6fc6VumXwV88N6oDgtOYQVBBwwGS+AI0AScYGIb/SnvxjJk750DpN9mAU3UgjXpHjYp/BTmEDt7Z6Mng4jaOmmDTLALw2dgMixSi77CddOjlZ8bFhRjcz1tWoV06s0vkpcOJQ2/mtMj2FKOz9N0s0ZHh21Ydpa055WPwxu0/K/HcK4AGVVVA2suz503TGDfVnm6VzjxMqKTGUmHN1WCIRBNalSSFMNSLbev39CJ1ewx6MM7ECpcdQbOseXdnu60uc/J2MF5So/5G3TMLOuLdo0uUvnQOvOv1FUTYJgyhoxd+I0PYuegs5csXPC02swEI5ovSb48+s7RFPWaf/GVn3n/hvF/dwnCxTYUpFpVCzZFquhxuxyN5h9Y+Toy8UK64yRkR9wcFy3P9h+72QRmHt8TeI2qW0LGOz7+NlVPdZd1H1s2eOed/7l3Fxl0Jv7DU/H6rGFcM2tLFuy50= ceph-40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:05:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.085+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1bd402fd50 msgr2=0x7f1bd4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.085+0000 7f1bfa1b7700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1bd402fd50 0x7f1bd4032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1be40096a0 tx=0x7f1be4006b40).stop 2023-12-08T10:05:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.085+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 msgr2=0x7f1bf410cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:05:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.085+0000 7f1bfa1b7700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf410cbc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1bdc00b190 tx=0x7f1bdc006ee0).stop 2023-12-08T10:05:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.086+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 shutdown_connections 2023-12-08T10:05:58.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.086+0000 7f1bfa1b7700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1bd402fd50 0x7f1bd4032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.086+0000 7f1bfa1b7700 1 --2- 172.21.15.23:0/1375400730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bf40703f0 0x7f1bf410cbc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:05:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.086+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 >> 172.21.15.23:0/1375400730 conn(0x7f1bf40f63c0 msgr2=0x7f1bf40f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:05:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.086+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 shutdown_connections 2023-12-08T10:05:58.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:05:58.086+0000 7f1bfa1b7700 1 -- 172.21.15.23:0/1375400730 wait complete. 2023-12-08T10:05:59.185 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:05:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:05:58.085288+0000 mgr.smithi023.tgiigg (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:05:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-08T10:05:59.994 INFO:teuthology.orchestra.run.smithi023.stderr:Adding key to root@localhost authorized_keys... 2023-12-08T10:05:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:Adding host smithi023... 2023-12-08T10:05:59.995 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi023 172.21.15.23 2023-12-08T10:06:00.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.370+0000 7f42ca3ba700 1 Processor -- start 2023-12-08T10:06:00.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.370+0000 7f42ca3ba700 1 -- start start 2023-12-08T10:06:00.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.370+0000 7f42ca3ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c40fe800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:00.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.370+0000 7f42ca3ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c40fecf0 con 0x7f42c4070430 2023-12-08T10:06:00.375 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c40fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c40fe800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53638/0 (socket says 172.21.15.23:53638) 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c3fff700 1 -- 172.21.15.23:0/1179210394 learned_addr learned my addr 172.21.15.23:0/1179210394 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c3fff700 1 -- 172.21.15.23:0/1179210394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c40fee30 con 0x7f42c4070430 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c3fff700 1 --2- 172.21.15.23:0/1179210394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c40fe800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f42b400b5f0 tx=0x7f42b4006ee0).ready entity=mon.0 client_cookie=fa0c5a6499da750 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c2ffd700 1 -- 172.21.15.23:0/1179210394 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42b400d600 con 0x7f42c4070430 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c2ffd700 1 -- 172.21.15.23:0/1179210394 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f42b400dbd0 con 0x7f42c4070430 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.371+0000 7f42c2ffd700 1 -- 172.21.15.23:0/1179210394 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42b4011c00 con 0x7f42c4070430 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 -- 172.21.15.23:0/1179210394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 msgr2=0x7f42c40fe800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 --2- 172.21.15.23:0/1179210394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c40fe800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f42b400b5f0 tx=0x7f42b4006ee0).stop 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 -- 172.21.15.23:0/1179210394 shutdown_connections 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 --2- 172.21.15.23:0/1179210394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c40fe800 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 -- 172.21.15.23:0/1179210394 >> 172.21.15.23:0/1179210394 conn(0x7f42c40f62a0 msgr2=0x7f42c40f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 -- 172.21.15.23:0/1179210394 shutdown_connections 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 -- 172.21.15.23:0/1179210394 wait complete. 2023-12-08T10:06:00.376 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.372+0000 7f42ca3ba700 1 Processor -- start 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42ca3ba700 1 -- start start 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42ca3ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c410a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42ca3ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c410aec0 con 0x7f42c4070430 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c410a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c410a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53644/0 (socket says 172.21.15.23:53644) 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42c3fff700 1 -- 172.21.15.23:0/2968774890 learned_addr learned my addr 172.21.15.23:0/2968774890 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42c3fff700 1 -- 172.21.15.23:0/2968774890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b400b200 con 0x7f42c4070430 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42c3fff700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c410a9d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f42b4002f40 tx=0x7f42b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.373+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42b4011690 con 0x7f42c4070430 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.374+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f42b401b880 con 0x7f42c4070430 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.374+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c410b0c0 con 0x7f42c4070430 2023-12-08T10:06:00.377 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.374+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42b401a6b0 con 0x7f42c4070430 2023-12-08T10:06:00.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.374+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c410b550 con 0x7f42c4070430 2023-12-08T10:06:00.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.374+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f42b400f070 con 0x7f42c4070430 2023-12-08T10:06:00.378 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.374+0000 7f42c0ff9700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f42b002fdb0 0x7f42b0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:00.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.375+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f42b4041120 con 0x7f42c4070430 2023-12-08T10:06:00.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.375+0000 7f42c37fe700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f42b002fdb0 0x7f42b0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:00.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.375+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c4045ad0 con 0x7f42c4070430 2023-12-08T10:06:00.379 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.375+0000 7f42c37fe700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f42b002fdb0 0x7f42b0032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f42b80096a0 tx=0x7f42b8006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:00.380 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.379+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42b4016070 con 0x7f42c4070430 2023-12-08T10:06:00.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:00.559+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7f42c41048e0 con 0x7f42b002fdb0 2023-12-08T10:06:01.161 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.159+0000 7f42c0ff9700 1 -- 172.21.15.23:0/2968774890 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f42c41048e0 con 0x7f42b002fdb0 2023-12-08T10:06:01.161 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Added host 'smithi023' 2023-12-08T10:06:01.163 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.161+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f42b002fdb0 msgr2=0x7f42b0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:01.163 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.161+0000 7f42ca3ba700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f42b002fdb0 0x7f42b0032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f42b80096a0 tx=0x7f42b8006b40).stop 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.161+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 msgr2=0x7f42c410a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.161+0000 7f42ca3ba700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c410a9d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f42b4002f40 tx=0x7f42b4006ee0).stop 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.162+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 shutdown_connections 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.162+0000 7f42ca3ba700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f42b002fdb0 0x7f42b0032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.162+0000 7f42ca3ba700 1 --2- 172.21.15.23:0/2968774890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42c4070430 0x7f42c410a9d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.162+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 >> 172.21.15.23:0/2968774890 conn(0x7f42c40f62a0 msgr2=0x7f42c40f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.162+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 shutdown_connections 2023-12-08T10:06:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.162+0000 7f42ca3ba700 1 -- 172.21.15.23:0/2968774890 wait complete. 2023-12-08T10:06:01.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:00 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:00.560397+0000 mgr.smithi023.tgiigg (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:01.457 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying mon service with default placement... 2023-12-08T10:06:01.457 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-12-08T10:06:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.827+0000 7f37fa8cc700 1 Processor -- start 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.827+0000 7f37fa8cc700 1 -- start start 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.827+0000 7f37fa8cc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.827+0000 7f37fa8cc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f4070790 con 0x7f37f4070370 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53656/0 (socket says 172.21.15.23:53656) 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f3fff700 1 -- 172.21.15.23:0/2395896839 learned_addr learned my addr 172.21.15.23:0/2395896839 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f3fff700 1 -- 172.21.15.23:0/2395896839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f40fee90 con 0x7f37f4070370 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f3fff700 1 --2- 172.21.15.23:0/2395896839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fe9a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f37e800b5f0 tx=0x7f37e8006ee0).ready entity=mon.0 client_cookie=32e5f019e0556d06 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f2ffd700 1 -- 172.21.15.23:0/2395896839 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e800d600 con 0x7f37f4070370 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f2ffd700 1 -- 172.21.15.23:0/2395896839 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f37e800dbd0 con 0x7f37f4070370 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.828+0000 7f37f2ffd700 1 -- 172.21.15.23:0/2395896839 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e8011c00 con 0x7f37f4070370 2023-12-08T10:06:01.831 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 -- 172.21.15.23:0/2395896839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 msgr2=0x7f37f40fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 --2- 172.21.15.23:0/2395896839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fe9a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f37e800b5f0 tx=0x7f37e8006ee0).stop 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 -- 172.21.15.23:0/2395896839 shutdown_connections 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 --2- 172.21.15.23:0/2395896839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fe9a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 -- 172.21.15.23:0/2395896839 >> 172.21.15.23:0/2395896839 conn(0x7f37f40f63c0 msgr2=0x7f37f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 -- 172.21.15.23:0/2395896839 shutdown_connections 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.829+0000 7f37fa8cc700 1 -- 172.21.15.23:0/2395896839 wait complete. 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37fa8cc700 1 Processor -- start 2023-12-08T10:06:01.832 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37fa8cc700 1 -- start start 2023-12-08T10:06:01.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37fa8cc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37fa8cc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40fa8a0 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53672/0 (socket says 172.21.15.23:53672) 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37f3fff700 1 -- 172.21.15.23:0/955683304 learned_addr learned my addr 172.21.15.23:0/955683304 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.830+0000 7f37f3fff700 1 -- 172.21.15.23:0/955683304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e800b200 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.831+0000 7f37f3fff700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fc420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f37e800b190 tx=0x7f37e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.831+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e801b410 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.831+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f40faa40 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.831+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f37e801b9e0 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.831+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e801a8f0 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.831+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f40fae50 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.832+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f37e800f070 con 0x7f37f4070370 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.832+0000 7f37f0ff9700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f37e002fd50 0x7f37e0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:01.834 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.832+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f37e801bb50 con 0x7f37f4070370 2023-12-08T10:06:01.835 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.832+0000 7f37f37fe700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f37e002fd50 0x7f37e0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:01.835 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.832+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4045ad0 con 0x7f37f4070370 2023-12-08T10:06:01.835 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.832+0000 7f37f37fe700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f37e002fd50 0x7f37e0032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f37e40096a0 tx=0x7f37e4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:01.837 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:01.836+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37e8016030 con 0x7f37f4070370 2023-12-08T10:06:02.025 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.024+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f37f40008d0 con 0x7f37e002fd50 2023-12-08T10:06:02.031 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.030+0000 7f37f0ff9700 1 -- 172.21.15.23:0/955683304 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f37f40008d0 con 0x7f37e002fd50 2023-12-08T10:06:02.031 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-08T10:06:02.033 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.031+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f37e002fd50 msgr2=0x7f37e0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.031+0000 7f37fa8cc700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f37e002fd50 0x7f37e0032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f37e40096a0 tx=0x7f37e4006b40).stop 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.031+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 msgr2=0x7f37f40fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.031+0000 7f37fa8cc700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fc420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f37e800b190 tx=0x7f37e8006ee0).stop 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.031+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 shutdown_connections 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.031+0000 7f37fa8cc700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f37e002fd50 0x7f37e0032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.032+0000 7f37fa8cc700 1 --2- 172.21.15.23:0/955683304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37f4070370 0x7f37f40fc420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.032+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 >> 172.21.15.23:0/955683304 conn(0x7f37f40f63c0 msgr2=0x7f37f40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.032+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 shutdown_connections 2023-12-08T10:06:02.034 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.032+0000 7f37fa8cc700 1 -- 172.21.15.23:0/955683304 wait complete. 2023-12-08T10:06:02.368 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying mgr service with default placement... 2023-12-08T10:06:02.368 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-12-08T10:06:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:01.159271+0000 mon.smithi023 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:01.159593+0000 mgr.smithi023.tgiigg (mgr.14116) 9 : cephadm [INF] Added host smithi023 2023-12-08T10:06:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:01.159836+0000 mon.smithi023 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:06:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:02.030149+0000 mon.smithi023 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:02.732 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.730+0000 7f1d1d8b5700 1 Processor -- start 2023-12-08T10:06:02.733 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.730+0000 7f1d1d8b5700 1 -- start start 2023-12-08T10:06:02.733 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d1d8b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:02.733 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d1d8b5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d180fedd0 con 0x7f1d18070430 2023-12-08T10:06:02.733 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d16ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d16ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53674/0 (socket says 172.21.15.23:53674) 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d16ffd700 1 -- 172.21.15.23:0/2754168807 learned_addr learned my addr 172.21.15.23:0/2754168807 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d16ffd700 1 -- 172.21.15.23:0/2754168807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d180fef10 con 0x7f1d18070430 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d16ffd700 1 --2- 172.21.15.23:0/2754168807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fe8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f1d0800b5f0 tx=0x7f1d08006ee0).ready entity=mon.0 client_cookie=88d0ca2063cb66c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d15ffb700 1 -- 172.21.15.23:0/2754168807 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d0800d600 con 0x7f1d18070430 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d15ffb700 1 -- 172.21.15.23:0/2754168807 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1d0800dbd0 con 0x7f1d18070430 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.731+0000 7f1d15ffb700 1 -- 172.21.15.23:0/2754168807 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d08011c00 con 0x7f1d18070430 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/2754168807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 msgr2=0x7f1d180fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:02.736 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 --2- 172.21.15.23:0/2754168807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fe8e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f1d0800b5f0 tx=0x7f1d08006ee0).stop 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/2754168807 shutdown_connections 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 --2- 172.21.15.23:0/2754168807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fe8e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/2754168807 >> 172.21.15.23:0/2754168807 conn(0x7f1d180f63c0 msgr2=0x7f1d180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/2754168807 shutdown_connections 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.732+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/2754168807 wait complete. 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d1d8b5700 1 Processor -- start 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d1d8b5700 1 -- start start 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d1d8b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d1d8b5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d180fa7e0 con 0x7f1d18070430 2023-12-08T10:06:02.737 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d16ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:02.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d16ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53684/0 (socket says 172.21.15.23:53684) 2023-12-08T10:06:02.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.733+0000 7f1d16ffd700 1 -- 172.21.15.23:0/3276908268 learned_addr learned my addr 172.21.15.23:0/3276908268 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:02.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1d16ffd700 1 -- 172.21.15.23:0/3276908268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0800b200 con 0x7f1d18070430 2023-12-08T10:06:02.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1d16ffd700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fc360 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1d0800b190 tx=0x7f1d08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:02.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d0801b410 con 0x7f1d18070430 2023-12-08T10:06:02.739 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1d0801b9e0 con 0x7f1d18070430 2023-12-08T10:06:02.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d180fa980 con 0x7f1d18070430 2023-12-08T10:06:02.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d0801a880 con 0x7f1d18070430 2023-12-08T10:06:02.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.734+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d180fad90 con 0x7f1d18070430 2023-12-08T10:06:02.740 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.735+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d18045ad0 con 0x7f1d18070430 2023-12-08T10:06:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.736+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1d0800f070 con 0x7f1d18070430 2023-12-08T10:06:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.736+0000 7f1cff7fe700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1d0402fdb0 0x7f1d04032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.736+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1d08041d80 con 0x7f1d18070430 2023-12-08T10:06:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.736+0000 7f1d167fc700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1d0402fdb0 0x7f1d04032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.736+0000 7f1d167fc700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1d0402fdb0 0x7f1d04032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f1d0c0096a0 tx=0x7f1d0c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:02.741 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.740+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d0802db70 con 0x7f1d18070430 2023-12-08T10:06:02.930 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.929+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d180fafd0 con 0x7f1d0402fdb0 2023-12-08T10:06:02.936 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.935+0000 7f1cff7fe700 1 -- 172.21.15.23:0/3276908268 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1d180fafd0 con 0x7f1d0402fdb0 2023-12-08T10:06:02.936 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-08T10:06:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1d0402fdb0 msgr2=0x7f1d04032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1d0402fdb0 0x7f1d04032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f1d0c0096a0 tx=0x7f1d0c006b40).stop 2023-12-08T10:06:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 msgr2=0x7f1d180fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fc360 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1d0800b190 tx=0x7f1d08006ee0).stop 2023-12-08T10:06:02.938 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 shutdown_connections 2023-12-08T10:06:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f1d0402fdb0 0x7f1d04032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 --2- 172.21.15.23:0/3276908268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d18070430 0x7f1d180fc360 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 >> 172.21.15.23:0/3276908268 conn(0x7f1d180f63c0 msgr2=0x7f1d180f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.936+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 shutdown_connections 2023-12-08T10:06:02.939 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:02.937+0000 7f1d1d8b5700 1 -- 172.21.15.23:0/3276908268 wait complete. 2023-12-08T10:06:03.277 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying crash service with default placement... 2023-12-08T10:06:03.277 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-12-08T10:06:03.630 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561f414700 1 Processor -- start 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561f414700 1 -- start start 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561f414700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f5618100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561f414700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56180fc730 con 0x7f56180703d0 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561d1b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f5618100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561d1b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f5618100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53696/0 (socket says 172.21.15.23:53696) 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.624+0000 7f561d1b0700 1 -- 172.21.15.23:0/4181641005 learned_addr learned my addr 172.21.15.23:0/4181641005 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.625+0000 7f561d1b0700 1 -- 172.21.15.23:0/4181641005 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5618101230 con 0x7f56180703d0 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.625+0000 7f561d1b0700 1 --2- 172.21.15.23:0/4181641005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f5618100d40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f560c00b5f0 tx=0x7f560c006ee0).ready entity=mon.0 client_cookie=786af90a0c5ddea7 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.625+0000 7f560bfff700 1 -- 172.21.15.23:0/4181641005 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560c00d600 con 0x7f56180703d0 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.625+0000 7f560bfff700 1 -- 172.21.15.23:0/4181641005 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f560c00dbd0 con 0x7f56180703d0 2023-12-08T10:06:03.631 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.625+0000 7f560bfff700 1 -- 172.21.15.23:0/4181641005 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560c011c00 con 0x7f56180703d0 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 -- 172.21.15.23:0/4181641005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 msgr2=0x7f5618100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 --2- 172.21.15.23:0/4181641005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f5618100d40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f560c00b5f0 tx=0x7f560c006ee0).stop 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 -- 172.21.15.23:0/4181641005 shutdown_connections 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 --2- 172.21.15.23:0/4181641005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f5618100d40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 -- 172.21.15.23:0/4181641005 >> 172.21.15.23:0/4181641005 conn(0x7f56180f63c0 msgr2=0x7f56180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 -- 172.21.15.23:0/4181641005 shutdown_connections 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 -- 172.21.15.23:0/4181641005 wait complete. 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 Processor -- start 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.626+0000 7f561f414700 1 -- start start 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561f414700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f561810cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:03.632 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561f414700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561810d100 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561d1b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f561810cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561d1b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f561810cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53704/0 (socket says 172.21.15.23:53704) 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561d1b0700 1 -- 172.21.15.23:0/1712193858 learned_addr learned my addr 172.21.15.23:0/1712193858 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561d1b0700 1 -- 172.21.15.23:0/1712193858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c00b200 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561d1b0700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f561810cc10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f560c00e040 tx=0x7f560c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560c00d9f0 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f560c01a9f0 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561810d300 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f560c01b8c0 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.627+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561810d710 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.628+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f560c00f070 con 0x7f56180703d0 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.628+0000 7f5609ffb700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f560002fdb0 0x7f5600032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:03.633 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.628+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f560c0420c0 con 0x7f56180703d0 2023-12-08T10:06:03.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.628+0000 7f561c9af700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f560002fdb0 0x7f5600032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:03.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.629+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f8004fa0 con 0x7f56180703d0 2023-12-08T10:06:03.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.629+0000 7f561c9af700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f560002fdb0 0x7f5600032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f56140096a0 tx=0x7f5614006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:03.634 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.633+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f560c016070 con 0x7f56180703d0 2023-12-08T10:06:03.822 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.820+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f55f8002a50 con 0x7f560002fdb0 2023-12-08T10:06:03.825 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.824+0000 7f5609ffb700 1 -- 172.21.15.23:0/1712193858 <== mgr.14116 v2:172.21.15.23:6800/4062254273 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f55f8002a50 con 0x7f560002fdb0 2023-12-08T10:06:03.825 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-08T10:06:03.827 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f560002fdb0 msgr2=0x7f5600032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f560002fdb0 0x7f5600032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f56140096a0 tx=0x7f5614006b40).stop 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 msgr2=0x7f561810cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f561810cc10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f560c00e040 tx=0x7f560c006ee0).stop 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 shutdown_connections 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f560002fdb0 0x7f5600032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 --2- 172.21.15.23:0/1712193858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56180703d0 0x7f561810cc10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 >> 172.21.15.23:0/1712193858 conn(0x7f56180f63c0 msgr2=0x7f56180f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 shutdown_connections 2023-12-08T10:06:03.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:03.826+0000 7f561f414700 1 -- 172.21.15.23:0/1712193858 wait complete. 2023-12-08T10:06:04.078 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:02.025142+0000 mgr.smithi023.tgiigg (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:04.079 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:02.025907+0000 mgr.smithi023.tgiigg (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-08T10:06:04.079 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:02.934940+0000 mon.smithi023 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:04.160 INFO:teuthology.orchestra.run.smithi023.stderr:Enabling mgr prometheus module... 2023-12-08T10:06:04.160 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.497+0000 7f75b5d24700 1 Processor -- start 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.497+0000 7f75b5d24700 1 -- start start 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.497+0000 7f75b5d24700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.497+0000 7f75b5d24700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00fee30 con 0x7f75b00703d0 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75af7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75af7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53714/0 (socket says 172.21.15.23:53714) 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75af7fe700 1 -- 172.21.15.23:0/1241429374 learned_addr learned my addr 172.21.15.23:0/1241429374 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75af7fe700 1 -- 172.21.15.23:0/1241429374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00fef70 con 0x7f75b00703d0 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75af7fe700 1 --2- 172.21.15.23:0/1241429374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fe940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f75a400b5f0 tx=0x7f75a4006ee0).ready entity=mon.0 client_cookie=35c7703c9a162173 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75ae7fc700 1 -- 172.21.15.23:0/1241429374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a400d600 con 0x7f75b00703d0 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75ae7fc700 1 -- 172.21.15.23:0/1241429374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f75a400dbd0 con 0x7f75b00703d0 2023-12-08T10:06:04.500 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.498+0000 7f75ae7fc700 1 -- 172.21.15.23:0/1241429374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a4011c00 con 0x7f75b00703d0 2023-12-08T10:06:04.504 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 -- 172.21.15.23:0/1241429374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 msgr2=0x7f75b00fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 --2- 172.21.15.23:0/1241429374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fe940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f75a400b5f0 tx=0x7f75a4006ee0).stop 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 -- 172.21.15.23:0/1241429374 shutdown_connections 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 --2- 172.21.15.23:0/1241429374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fe940 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 -- 172.21.15.23:0/1241429374 >> 172.21.15.23:0/1241429374 conn(0x7f75b00f63c0 msgr2=0x7f75b00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 -- 172.21.15.23:0/1241429374 shutdown_connections 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 -- 172.21.15.23:0/1241429374 wait complete. 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.499+0000 7f75b5d24700 1 Processor -- start 2023-12-08T10:06:04.505 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75b5d24700 1 -- start start 2023-12-08T10:06:04.506 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75b5d24700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:04.506 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75b5d24700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b00fa840 con 0x7f75b00703d0 2023-12-08T10:06:04.506 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75af7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:04.506 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75af7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53728/0 (socket says 172.21.15.23:53728) 2023-12-08T10:06:04.506 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75af7fe700 1 -- 172.21.15.23:0/2789426546 learned_addr learned my addr 172.21.15.23:0/2789426546 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75af7fe700 1 -- 172.21.15.23:0/2789426546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a400b200 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75af7fe700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fc3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f75a400b190 tx=0x7f75a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a401b410 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b00fa9e0 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f75a401b9e0 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.500+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75a401a880 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.501+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b00fadf0 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.501+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f75a400f070 con 0x7f75b00703d0 2023-12-08T10:06:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.501+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b0045ad0 con 0x7f75b00703d0 2023-12-08T10:06:04.508 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.501+0000 7f7597fff700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f759c02fd50 0x7f759c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:04.508 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.502+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f75a4041280 con 0x7f75b00703d0 2023-12-08T10:06:04.508 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.502+0000 7f75aeffd700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f759c02fd50 0x7f759c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:04.508 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.502+0000 7f75aeffd700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f759c02fd50 0x7f759c032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f75a00096a0 tx=0x7f75a0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:04.509 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.506+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75a4022ac0 con 0x7f75b00703d0 2023-12-08T10:06:04.703 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.702+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f75b00ffb90 con 0x7f75b00703d0 2023-12-08T10:06:04.828 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.826+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f75a4025980 con 0x7f75b00703d0 2023-12-08T10:06:04.829 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.828+0000 7f7597fff700 1 -- 172.21.15.23:0/2789426546 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f75a40197b0 con 0x7f75b00703d0 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f759c02fd50 msgr2=0x7f759c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f759c02fd50 0x7f759c032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f75a00096a0 tx=0x7f75a0006b40).stop 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 msgr2=0x7f75b00fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fc3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f75a400b190 tx=0x7f75a4006ee0).stop 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 shutdown_connections 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f759c02fd50 0x7f759c032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 --2- 172.21.15.23:0/2789426546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75b00703d0 0x7f75b00fc3c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.830+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 >> 172.21.15.23:0/2789426546 conn(0x7f75b00f63c0 msgr2=0x7f75b00f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.831+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 shutdown_connections 2023-12-08T10:06:04.833 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:04.831+0000 7f75b5d24700 1 -- 172.21.15.23:0/2789426546 wait complete. 2023-12-08T10:06:05.153 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:02.930383+0000 mgr.smithi023.tgiigg (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:05.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:02.930874+0000 mgr.smithi023.tgiigg (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-08T10:06:05.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:03.821870+0000 mgr.smithi023.tgiigg (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:05.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:03.822478+0000 mgr.smithi023.tgiigg (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-08T10:06:05.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:03.824497+0000 mon.smithi023 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.23:0/2498343969' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:05.154 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:04.702554+0000 mon.smithi023 (mon.0) 60 : audit [INF] from='client.? 172.21.15.23:0/2789426546' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-08T10:06:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying prometheus service with default placement... 2023-12-08T10:06:05.185 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-12-08T10:06:05.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.602+0000 7f72173cf700 1 Processor -- start 2023-12-08T10:06:05.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f72173cf700 1 -- start start 2023-12-08T10:06:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f72173cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f72100fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f72173cf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100fee10 con 0x7f72100703f0 2023-12-08T10:06:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f721516b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f72100fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:05.605 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f721516b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f72100fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53752/0 (socket says 172.21.15.23:53752) 2023-12-08T10:06:05.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f721516b700 1 -- 172.21.15.23:0/173523304 learned_addr learned my addr 172.21.15.23:0/173523304 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:05.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f721516b700 1 -- 172.21.15.23:0/173523304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100fef50 con 0x7f72100703f0 2023-12-08T10:06:05.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.603+0000 7f721516b700 1 --2- 172.21.15.23:0/173523304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f72100fe920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f720c00b5f0 tx=0x7f720c006ee0).ready entity=mon.0 client_cookie=21c60e659c9ebfc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:05.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.604+0000 7f71fffff700 1 -- 172.21.15.23:0/173523304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f720c00d600 con 0x7f72100703f0 2023-12-08T10:06:05.606 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.604+0000 7f71fffff700 1 -- 172.21.15.23:0/173523304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f720c00dbd0 con 0x7f72100703f0 2023-12-08T10:06:05.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.604+0000 7f71fffff700 1 -- 172.21.15.23:0/173523304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f720c011c00 con 0x7f72100703f0 2023-12-08T10:06:05.607 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 -- 172.21.15.23:0/173523304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 msgr2=0x7f72100fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 --2- 172.21.15.23:0/173523304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f72100fe920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f720c00b5f0 tx=0x7f720c006ee0).stop 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 -- 172.21.15.23:0/173523304 shutdown_connections 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 --2- 172.21.15.23:0/173523304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f72100fe920 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 -- 172.21.15.23:0/173523304 >> 172.21.15.23:0/173523304 conn(0x7f72100f63c0 msgr2=0x7f72100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 -- 172.21.15.23:0/173523304 shutdown_connections 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 -- 172.21.15.23:0/173523304 wait complete. 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 Processor -- start 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.605+0000 7f72173cf700 1 -- start start 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f72173cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f721010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f72173cf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721010af40 con 0x7f72100703f0 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f721516b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f721010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f721516b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f721010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53756/0 (socket says 172.21.15.23:53756) 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f721516b700 1 -- 172.21.15.23:0/2634573310 learned_addr learned my addr 172.21.15.23:0/2634573310 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f721516b700 1 -- 172.21.15.23:0/2634573310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c00b200 con 0x7f72100703f0 2023-12-08T10:06:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f721516b700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f721010aa50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f720c00e040 tx=0x7f720c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f720c00d9f0 con 0x7f72100703f0 2023-12-08T10:06:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721010b140 con 0x7f72100703f0 2023-12-08T10:06:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f720c01a9f0 con 0x7f72100703f0 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.606+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f720c01b8c0 con 0x7f72100703f0 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.607+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721010b550 con 0x7f72100703f0 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.608+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7210045ad0 con 0x7f72100703f0 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.608+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f720c00f070 con 0x7f72100703f0 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.608+0000 7f71fdffb700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.608+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f720c041380 con 0x7f72100703f0 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.608+0000 7f721496a700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 msgr2=0x7f71f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4062254273 2023-12-08T10:06:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.608+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-08T10:06:05.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.612+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f720c022d70 con 0x7f72100703f0 2023-12-08T10:06:05.800 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.799+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 --> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f721010be90 con 0x7f71f802fdb0 2023-12-08T10:06:05.810 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.809+0000 7f721496a700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 msgr2=0x7f71f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4062254273 2023-12-08T10:06:05.810 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:05.809+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-08T10:06:06.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:04.828434+0000 mon.smithi023 (mon.0) 61 : audit [INF] from='client.? 172.21.15.23:0/2789426546' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-08T10:06:06.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:04.828481+0000 mon.smithi023 (mon.0) 62 : cluster [DBG] mgrmap e9: smithi023.tgiigg(active, since 14s) 2023-12-08T10:06:06.211 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:06.210+0000 7f721496a700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 msgr2=0x7f71f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4062254273 2023-12-08T10:06:06.211 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:06.210+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-08T10:06:07.012 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:07.011+0000 7f721496a700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 msgr2=0x7f71f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4062254273 2023-12-08T10:06:07.012 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:07.011+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-08T10:06:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:08.612+0000 7f721496a700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 msgr2=0x7f71f8032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4062254273 2023-12-08T10:06:08.613 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:08.612+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-08T10:06:10.071 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:10.070+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f720c0257e0 con 0x7f72100703f0 2023-12-08T10:06:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:10.070+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 msgr2=0x7f71f8032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-08T10:06:10.072 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:10.070+0000 7f71fdffb700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:06:10.324 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:10.018228+0000 mon.smithi023 (mon.0) 63 : cluster [INF] Active manager daemon smithi023.tgiigg restarted 2023-12-08T10:06:10.325 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:10.018797+0000 mon.smithi023 (mon.0) 64 : cluster [INF] Activating manager daemon smithi023.tgiigg 2023-12-08T10:06:10.325 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:10.029783+0000 mon.smithi023 (mon.0) 65 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-08T10:06:11.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.083+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f720c03d1b0 con 0x7f72100703f0 2023-12-08T10:06:11.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.083+0000 7f71fdffb700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f71f8033c20 0x7f71f8036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:11.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.083+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 --> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f721010be90 con 0x7f71f8033c20 2023-12-08T10:06:11.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.083+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f71f8033c20 0x7f71f8036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:11.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.084+0000 7f721496a700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f71f8033c20 0x7f71f8036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f72000036d0 tx=0x7f7200006cb0).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:11.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.093+0000 7f71fdffb700 1 -- 172.21.15.23:0/2634573310 <== mgr.14144 v2:172.21.15.23:6800/3181112905 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f721010be90 con 0x7f71f8033c20 2023-12-08T10:06:11.094 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f71f8033c20 msgr2=0x7f71f8036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f71f8033c20 0x7f71f8036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f72000036d0 tx=0x7f7200006cb0).stop 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 msgr2=0x7f721010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f721010aa50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f720c00e040 tx=0x7f720c006ee0).stop 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 shutdown_connections 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f71f8033c20 0x7f71f8036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:11.097 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72100703f0 0x7f721010aa50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 --2- 172.21.15.23:0/2634573310 >> [v2:172.21.15.23:6800/4062254273,v1:172.21.15.23:6801/4062254273] conn(0x7f71f802fdb0 0x7f71f8032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:06:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 >> 172.21.15.23:0/2634573310 conn(0x7f72100f63c0 msgr2=0x7f72100f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 shutdown_connections 2023-12-08T10:06:11.098 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.095+0000 7f72173cf700 1 -- 172.21.15.23:0/2634573310 wait complete. 2023-12-08T10:06:11.369 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:10.071297+0000 mon.smithi023 (mon.0) 66 : cluster [DBG] mgrmap e10: smithi023.tgiigg(active, starting, since 0.0525764s) 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.074067+0000 mon.smithi023 (mon.0) 67 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.074625+0000 mon.smithi023 (mon.0) 68 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mgr metadata", "who": "smithi023.tgiigg", "id": "smithi023.tgiigg"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.075653+0000 mon.smithi023 (mon.0) 69 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.075825+0000 mon.smithi023 (mon.0) 70 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.075969+0000 mon.smithi023 (mon.0) 71 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:10.089709+0000 mon.smithi023 (mon.0) 72 : cluster [INF] Manager daemon smithi023.tgiigg is now available 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.192549+0000 mon.smithi023 (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.194222+0000 mon.smithi023 (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.208975+0000 mon.smithi023 (mon.0) 75 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:06:11.370 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:10.211891+0000 mon.smithi023 (mon.0) 76 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:06:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying grafana service with default placement... 2023-12-08T10:06:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-12-08T10:06:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a8746700 1 Processor -- start 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a8746700 1 -- start start 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a8746700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a00fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a8746700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a00fede0 con 0x7f32a00704a0 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a64e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a00fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a64e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a00fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52880/0 (socket says 172.21.15.23:52880) 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.776+0000 7f32a64e2700 1 -- 172.21.15.23:0/2257041985 learned_addr learned my addr 172.21.15.23:0/2257041985 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a64e2700 1 -- 172.21.15.23:0/2257041985 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a00fef20 con 0x7f32a00704a0 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a64e2700 1 --2- 172.21.15.23:0/2257041985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a00fe8f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f329c00b5f0 tx=0x7f329c006ee0).ready entity=mon.0 client_cookie=ce3a61d27cf26260 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:11.779 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a54e0700 1 -- 172.21.15.23:0/2257041985 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329c00d600 con 0x7f32a00704a0 2023-12-08T10:06:11.780 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a54e0700 1 -- 172.21.15.23:0/2257041985 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f329c00dbd0 con 0x7f32a00704a0 2023-12-08T10:06:11.780 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a54e0700 1 -- 172.21.15.23:0/2257041985 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329c011c00 con 0x7f32a00704a0 2023-12-08T10:06:11.780 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a8746700 1 -- 172.21.15.23:0/2257041985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 msgr2=0x7f32a00fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:11.780 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.777+0000 7f32a8746700 1 --2- 172.21.15.23:0/2257041985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a00fe8f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f329c00b5f0 tx=0x7f329c006ee0).stop 2023-12-08T10:06:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.778+0000 7f32a8746700 1 -- 172.21.15.23:0/2257041985 shutdown_connections 2023-12-08T10:06:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.778+0000 7f32a8746700 1 --2- 172.21.15.23:0/2257041985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a00fe8f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.778+0000 7f32a8746700 1 -- 172.21.15.23:0/2257041985 >> 172.21.15.23:0/2257041985 conn(0x7f32a00f63b0 msgr2=0x7f32a00f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.778+0000 7f32a8746700 1 -- 172.21.15.23:0/2257041985 shutdown_connections 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.778+0000 7f32a8746700 1 -- 172.21.15.23:0/2257041985 wait complete. 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a8746700 1 Processor -- start 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a8746700 1 -- start start 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a8746700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a8746700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a010aed0 con 0x7f32a00704a0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a64e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a64e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52882/0 (socket says 172.21.15.23:52882) 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a64e2700 1 -- 172.21.15.23:0/36134394 learned_addr learned my addr 172.21.15.23:0/36134394 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a64e2700 1 -- 172.21.15.23:0/36134394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c00b200 con 0x7f32a00704a0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f32a64e2700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a010a9e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f329c00b190 tx=0x7f329c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.779+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329c01b410 con 0x7f32a00704a0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f329c01b9e0 con 0x7f32a00704a0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a010b0d0 con 0x7f32a00704a0 2023-12-08T10:06:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f329c01a880 con 0x7f32a00704a0 2023-12-08T10:06:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a010b4e0 con 0x7f32a00704a0 2023-12-08T10:06:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f329c00f070 con 0x7f32a00704a0 2023-12-08T10:06:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f328effd700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f328802fcf0 0x7f32880321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.780+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f329c0412a0 con 0x7f32a00704a0 2023-12-08T10:06:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.781+0000 7f32a5ce1700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f328802fcf0 0x7f32880321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.781+0000 7f32a5ce1700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f328802fcf0 0x7f32880321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f32900096a0 tx=0x7f3290006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.781+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a0059100 con 0x7f32a00704a0 2023-12-08T10:06:11.786 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.785+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f329c016070 con 0x7f32a00704a0 2023-12-08T10:06:11.973 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.971+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 --> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f32a0104900 con 0x7f328802fcf0 2023-12-08T10:06:11.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.980+0000 7f328effd700 1 -- 172.21.15.23:0/36134394 <== mgr.14144 v2:172.21.15.23:6800/3181112905 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f32a0104900 con 0x7f328802fcf0 2023-12-08T10:06:11.981 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-08T10:06:11.984 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f328802fcf0 msgr2=0x7f32880321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:11.984 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f328802fcf0 0x7f32880321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f32900096a0 tx=0x7f3290006b40).stop 2023-12-08T10:06:11.984 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 msgr2=0x7f32a010a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:11.984 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a010a9e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f329c00b190 tx=0x7f329c006ee0).stop 2023-12-08T10:06:11.984 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 shutdown_connections 2023-12-08T10:06:11.985 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f328802fcf0 0x7f32880321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:11.985 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 --2- 172.21.15.23:0/36134394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32a00704a0 0x7f32a010a9e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:11.985 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 >> 172.21.15.23:0/36134394 conn(0x7f32a00f63b0 msgr2=0x7f32a00f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:11.985 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 shutdown_connections 2023-12-08T10:06:11.985 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:11.982+0000 7f32a8746700 1 -- 172.21.15.23:0/36134394 wait complete. 2023-12-08T10:06:12.269 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:11.085122+0000 mon.smithi023 (mon.0) 77 : cluster [DBG] mgrmap e11: smithi023.tgiigg(active, since 1.0664s) 2023-12-08T10:06:12.269 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:11.086312+0000 mgr.smithi023.tgiigg (mgr.14144) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-08T10:06:12.269 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:11.093047+0000 mon.smithi023 (mon.0) 78 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:12.269 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:11.980446+0000 mon.smithi023 (mon.0) 79 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:12.341 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying node-exporter service with default placement... 2023-12-08T10:06:12.342 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53d137c700 1 Processor -- start 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53d137c700 1 -- start start 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53d137c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc0fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53d137c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc070790 con 0x7f53cc070370 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53cbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc0fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53cbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc0fe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52892/0 (socket says 172.21.15.23:52892) 2023-12-08T10:06:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53cbfff700 1 -- 172.21.15.23:0/283570594 learned_addr learned my addr 172.21.15.23:0/283570594 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53cbfff700 1 -- 172.21.15.23:0/283570594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc0fece0 con 0x7f53cc070370 2023-12-08T10:06:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.665+0000 7f53cbfff700 1 --2- 172.21.15.23:0/283570594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc0fe7f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f53bc00b5f0 tx=0x7f53bc006ee0).ready entity=mon.0 client_cookie=eed25a732d1ac559 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53caffd700 1 -- 172.21.15.23:0/283570594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc00d600 con 0x7f53cc070370 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53caffd700 1 -- 172.21.15.23:0/283570594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53bc00dbd0 con 0x7f53cc070370 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53caffd700 1 -- 172.21.15.23:0/283570594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc011c00 con 0x7f53cc070370 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 -- 172.21.15.23:0/283570594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 msgr2=0x7f53cc0fe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 --2- 172.21.15.23:0/283570594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc0fe7f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f53bc00b5f0 tx=0x7f53bc006ee0).stop 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 -- 172.21.15.23:0/283570594 shutdown_connections 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 --2- 172.21.15.23:0/283570594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc0fe7f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 -- 172.21.15.23:0/283570594 >> 172.21.15.23:0/283570594 conn(0x7f53cc0f6210 msgr2=0x7f53cc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 -- 172.21.15.23:0/283570594 shutdown_connections 2023-12-08T10:06:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.666+0000 7f53d137c700 1 -- 172.21.15.23:0/283570594 wait complete. 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53d137c700 1 Processor -- start 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53d137c700 1 -- start start 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53d137c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc10a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53d137c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53cc10acb0 con 0x7f53cc070370 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53cbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc10a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53cbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc10a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52906/0 (socket says 172.21.15.23:52906) 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.667+0000 7f53cbfff700 1 -- 172.21.15.23:0/3148065964 learned_addr learned my addr 172.21.15.23:0/3148065964 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53cbfff700 1 -- 172.21.15.23:0/3148065964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc00b200 con 0x7f53cc070370 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53cbfff700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc10a7c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f53bc00b190 tx=0x7f53bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc01b410 con 0x7f53cc070370 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f53bc01b9e0 con 0x7f53cc070370 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc10aeb0 con 0x7f53cc070370 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53bc01a8f0 con 0x7f53cc070370 2023-12-08T10:06:12.673 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.668+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc10b2c0 con 0x7f53cc070370 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.669+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f53bc00f070 con 0x7f53cc070370 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.669+0000 7f53c8ff9700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f53b802fcf0 0x7f53b80321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.669+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f53bc041380 con 0x7f53cc070370 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.669+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53cc045ad0 con 0x7f53cc070370 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.669+0000 7f53cb7fe700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f53b802fcf0 0x7f53b80321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.669+0000 7f53cb7fe700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f53b802fcf0 0x7f53b80321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f53c00096a0 tx=0x7f53c0006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:12.674 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.673+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53bc016070 con 0x7f53cc070370 2023-12-08T10:06:12.861 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.860+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 --> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f53cc10bba0 con 0x7f53b802fcf0 2023-12-08T10:06:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.869+0000 7f53c8ff9700 1 -- 172.21.15.23:0/3148065964 <== mgr.14144 v2:172.21.15.23:6800/3181112905 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f53cc10bba0 con 0x7f53b802fcf0 2023-12-08T10:06:12.870 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f53b802fcf0 msgr2=0x7f53b80321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f53b802fcf0 0x7f53b80321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f53c00096a0 tx=0x7f53c0006b40).stop 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 msgr2=0x7f53cc10a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc10a7c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f53bc00b190 tx=0x7f53bc006ee0).stop 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 shutdown_connections 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f53b802fcf0 0x7f53b80321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 --2- 172.21.15.23:0/3148065964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f53cc070370 0x7f53cc10a7c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 >> 172.21.15.23:0/3148065964 conn(0x7f53cc0f6210 msgr2=0x7f53cc0f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 shutdown_connections 2023-12-08T10:06:12.873 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:12.871+0000 7f53d137c700 1 -- 172.21.15.23:0/3148065964 wait complete. 2023-12-08T10:06:13.225 INFO:teuthology.orchestra.run.smithi023.stderr:Deploying alertmanager service with default placement... 2023-12-08T10:06:13.226 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-12-08T10:06:13.559 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baef37700 1 Processor -- start 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baef37700 1 -- start start 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baef37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba80fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baef37700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba80fb4a0 con 0x7f0ba80fab90 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baccd3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba80fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baccd3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba80fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52914/0 (socket says 172.21.15.23:52914) 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.557+0000 7f0baccd3700 1 -- 172.21.15.23:0/432801993 learned_addr learned my addr 172.21.15.23:0/432801993 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.558+0000 7f0baccd3700 1 -- 172.21.15.23:0/432801993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80fb5e0 con 0x7f0ba80fab90 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.558+0000 7f0baccd3700 1 --2- 172.21.15.23:0/432801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba80fafb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f0b9c00b870 tx=0x7f0b9c006f70).ready entity=mon.0 client_cookie=4c4e92db43ddd92c server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:13.560 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.558+0000 7f0ba77fe700 1 -- 172.21.15.23:0/432801993 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c00da20 con 0x7f0ba80fab90 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.558+0000 7f0ba77fe700 1 -- 172.21.15.23:0/432801993 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b9c010470 con 0x7f0ba80fab90 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.558+0000 7f0ba77fe700 1 -- 172.21.15.23:0/432801993 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c00f5b0 con 0x7f0ba80fab90 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 -- 172.21.15.23:0/432801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 msgr2=0x7f0ba80fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 --2- 172.21.15.23:0/432801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba80fafb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f0b9c00b870 tx=0x7f0b9c006f70).stop 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 -- 172.21.15.23:0/432801993 shutdown_connections 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 --2- 172.21.15.23:0/432801993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba80fafb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 -- 172.21.15.23:0/432801993 >> 172.21.15.23:0/432801993 conn(0x7f0ba80f63b0 msgr2=0x7f0ba80f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 -- 172.21.15.23:0/432801993 shutdown_connections 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.559+0000 7f0baef37700 1 -- 172.21.15.23:0/432801993 wait complete. 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baef37700 1 Processor -- start 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baef37700 1 -- start start 2023-12-08T10:06:13.562 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baef37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba8108730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:13.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baef37700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba8108c20 con 0x7f0ba80fab90 2023-12-08T10:06:13.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baccd3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba8108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:13.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baccd3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba8108730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52930/0 (socket says 172.21.15.23:52930) 2023-12-08T10:06:13.563 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baccd3700 1 -- 172.21.15.23:0/3316892281 learned_addr learned my addr 172.21.15.23:0/3316892281 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.560+0000 7f0baccd3700 1 -- 172.21.15.23:0/3316892281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b9c00b480 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.561+0000 7f0baccd3700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba8108730 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0b9c006a40 tx=0x7f0b9c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.561+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c00f3d0 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.561+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0b9c00fef0 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.561+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b9c019d40 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.561+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba8108e20 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.561+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba8109230 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.562+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0b9c02b060 con 0x7f0ba80fab90 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.562+0000 7f0ba57fa700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f0b8c02fe60 0x7f0b8c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:13.564 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.562+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0b9c044310 con 0x7f0ba80fab90 2023-12-08T10:06:13.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.562+0000 7f0ba7fff700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f0b8c02fe60 0x7f0b8c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:13.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.562+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba8059100 con 0x7f0ba80fab90 2023-12-08T10:06:13.565 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.562+0000 7f0ba7fff700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f0b8c02fe60 0x7f0b8c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0b940096a0 tx=0x7f0b94006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:13.567 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.567+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b9c014020 con 0x7f0ba80fab90 2023-12-08T10:06:13.754 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.753+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 --> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ba8102650 con 0x7f0b8c02fe60 2023-12-08T10:06:13.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.762+0000 7f0ba57fa700 1 -- 172.21.15.23:0/3316892281 <== mgr.14144 v2:172.21.15.23:6800/3181112905 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f0ba8102650 con 0x7f0b8c02fe60 2023-12-08T10:06:13.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-08T10:06:13.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f0b8c02fe60 msgr2=0x7f0b8c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f0b8c02fe60 0x7f0b8c032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0b940096a0 tx=0x7f0b94006b40).stop 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 msgr2=0x7f0ba8108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba8108730 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f0b9c006a40 tx=0x7f0b9c006f70).stop 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 shutdown_connections 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f0b8c02fe60 0x7f0b8c032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 --2- 172.21.15.23:0/3316892281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba80fab90 0x7f0ba8108730 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 >> 172.21.15.23:0/3316892281 conn(0x7f0ba80f63b0 msgr2=0x7f0ba80ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 shutdown_connections 2023-12-08T10:06:13.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:13.764+0000 7f0baef37700 1 -- 172.21.15.23:0/3316892281 wait complete. 2023-12-08T10:06:14.052 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:11.972822+0000 mgr.smithi023.tgiigg (mgr.14144) 3 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:14.052 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:11.973323+0000 mgr.smithi023.tgiigg (mgr.14144) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-08T10:06:14.052 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:12.861472+0000 mgr.smithi023.tgiigg (mgr.14144) 5 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:14.052 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:12.862136+0000 mgr.smithi023.tgiigg (mgr.14144) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-08T10:06:14.052 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:12.869436+0000 mon.smithi023 (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:14.053 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:12.986642+0000 mon.smithi023 (mon.0) 81 : cluster [DBG] mgrmap e12: smithi023.tgiigg(active, since 2s) 2023-12-08T10:06:14.053 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:13.762208+0000 mon.smithi023 (mon.0) 82 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:14.100 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-12-08T10:06:14.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.422+0000 7fbf41927700 1 Processor -- start 2023-12-08T10:06:14.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.422+0000 7fbf41927700 1 -- start start 2023-12-08T10:06:14.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.422+0000 7fbf41927700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:14.424 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.422+0000 7fbf41927700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0fc710 con 0x7fbf3c0703f0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52938/0 (socket says 172.21.15.23:52938) 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf3affd700 1 -- 172.21.15.23:0/363341360 learned_addr learned my addr 172.21.15.23:0/363341360 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf3affd700 1 -- 172.21.15.23:0/363341360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c101210 con 0x7fbf3c0703f0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf3affd700 1 --2- 172.21.15.23:0/363341360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c100d20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fbf2400b5f0 tx=0x7fbf24006ee0).ready entity=mon.0 client_cookie=7c477162fbae3757 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf39ffb700 1 -- 172.21.15.23:0/363341360 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf2400d600 con 0x7fbf3c0703f0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf39ffb700 1 -- 172.21.15.23:0/363341360 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbf2400dbd0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.423+0000 7fbf39ffb700 1 -- 172.21.15.23:0/363341360 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf24011c00 con 0x7fbf3c0703f0 2023-12-08T10:06:14.426 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.424+0000 7fbf41927700 1 -- 172.21.15.23:0/363341360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 msgr2=0x7fbf3c100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:14.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.424+0000 7fbf41927700 1 --2- 172.21.15.23:0/363341360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c100d20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fbf2400b5f0 tx=0x7fbf24006ee0).stop 2023-12-08T10:06:14.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.424+0000 7fbf41927700 1 -- 172.21.15.23:0/363341360 shutdown_connections 2023-12-08T10:06:14.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.424+0000 7fbf41927700 1 --2- 172.21.15.23:0/363341360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c100d20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:14.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.424+0000 7fbf41927700 1 -- 172.21.15.23:0/363341360 >> 172.21.15.23:0/363341360 conn(0x7fbf3c0f63c0 msgr2=0x7fbf3c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:14.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.425+0000 7fbf41927700 1 -- 172.21.15.23:0/363341360 shutdown_connections 2023-12-08T10:06:14.427 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.425+0000 7fbf41927700 1 -- 172.21.15.23:0/363341360 wait complete. 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.425+0000 7fbf41927700 1 Processor -- start 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.425+0000 7fbf41927700 1 -- start start 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf41927700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c10cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf41927700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c10d1d0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c10cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf3affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c10cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52944/0 (socket says 172.21.15.23:52944) 2023-12-08T10:06:14.428 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf3affd700 1 -- 172.21.15.23:0/3412638558 learned_addr learned my addr 172.21.15.23:0/3412638558 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:14.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf3affd700 1 -- 172.21.15.23:0/3412638558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2400b200 con 0x7fbf3c0703f0 2023-12-08T10:06:14.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf3affd700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c10cce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fbf2400e040 tx=0x7fbf24006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:14.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf2400d9f0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbf2401a9f0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.429 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c10d3d0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.426+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf2401b8c0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.427+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c10d7e0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.427+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c045ad0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.428+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbf2400f070 con 0x7fbf3c0703f0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.428+0000 7fbf337fe700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fbf1c02feb0 0x7fbf1c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.428+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbf24041b60 con 0x7fbf3c0703f0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.428+0000 7fbf3a7fc700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fbf1c02feb0 0x7fbf1c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:14.430 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.428+0000 7fbf3a7fc700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fbf1c02feb0 0x7fbf1c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fbf2c0096a0 tx=0x7fbf2c006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:14.433 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.432+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf240198f0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.592+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fbf3c101dd0 con 0x7fbf3c0703f0 2023-12-08T10:06:14.600 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.599+0000 7fbf337fe700 1 -- 172.21.15.23:0/3412638558 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fbf24019300 con 0x7fbf3c0703f0 2023-12-08T10:06:14.603 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.601+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fbf1c02feb0 msgr2=0x7fbf1c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:14.603 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.601+0000 7fbf41927700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fbf1c02feb0 0x7fbf1c032370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fbf2c0096a0 tx=0x7fbf2c006b40).stop 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.601+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 msgr2=0x7fbf3c10cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.601+0000 7fbf41927700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c10cce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fbf2400e040 tx=0x7fbf24006ee0).stop 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.602+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 shutdown_connections 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.602+0000 7fbf41927700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fbf1c02feb0 0x7fbf1c032370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.602+0000 7fbf41927700 1 --2- 172.21.15.23:0/3412638558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf3c0703f0 0x7fbf3c10cce0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.602+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 >> 172.21.15.23:0/3412638558 conn(0x7fbf3c0f63c0 msgr2=0x7fbf3c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.602+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 shutdown_connections 2023-12-08T10:06:14.604 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:14.602+0000 7fbf41927700 1 -- 172.21.15.23:0/3412638558 wait complete. 2023-12-08T10:06:14.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:13.754433+0000 mgr.smithi023.tgiigg (mgr.14144) 7 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:14.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:13.754990+0000 mgr.smithi023.tgiigg (mgr.14144) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-08T10:06:14.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:14.600014+0000 mon.smithi023 (mon.0) 83 : audit [INF] from='client.? 172.21.15.23:0/3412638558' entity='client.admin' 2023-12-08T10:06:14.983 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fd0051ff700 1 Processor -- start 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fd0051ff700 1 -- start start 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fd0051ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd0000fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fd0051ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000fb2e0 con 0x7fd0000fa9d0 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fcffffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd0000fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fcffffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd0000fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52948/0 (socket says 172.21.15.23:52948) 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.327+0000 7fcffffff700 1 -- 172.21.15.23:0/2663458537 learned_addr learned my addr 172.21.15.23:0/2663458537 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.328+0000 7fcffffff700 1 -- 172.21.15.23:0/2663458537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0000fb420 con 0x7fd0000fa9d0 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.328+0000 7fcffffff700 1 --2- 172.21.15.23:0/2663458537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd0000fadf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcff000b5f0 tx=0x7fcff0006ee0).ready entity=mon.0 client_cookie=b9049b2356bacdbc server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:15.332 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.328+0000 7fcffeffd700 1 -- 172.21.15.23:0/2663458537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff000d600 con 0x7fd0000fa9d0 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.328+0000 7fcffeffd700 1 -- 172.21.15.23:0/2663458537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcff000dbd0 con 0x7fd0000fa9d0 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.328+0000 7fcffeffd700 1 -- 172.21.15.23:0/2663458537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff0011c00 con 0x7fd0000fa9d0 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 -- 172.21.15.23:0/2663458537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 msgr2=0x7fd0000fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 --2- 172.21.15.23:0/2663458537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd0000fadf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcff000b5f0 tx=0x7fcff0006ee0).stop 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 -- 172.21.15.23:0/2663458537 shutdown_connections 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 --2- 172.21.15.23:0/2663458537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd0000fadf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 -- 172.21.15.23:0/2663458537 >> 172.21.15.23:0/2663458537 conn(0x7fd0000f6210 msgr2=0x7fd0000f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 -- 172.21.15.23:0/2663458537 shutdown_connections 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.329+0000 7fd0051ff700 1 -- 172.21.15.23:0/2663458537 wait complete. 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fd0051ff700 1 Processor -- start 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fd0051ff700 1 -- start start 2023-12-08T10:06:15.333 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fd0051ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd00010d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:15.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fd0051ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000070470 con 0x7fd0000fa9d0 2023-12-08T10:06:15.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fcffffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd00010d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:15.334 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fcffffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd00010d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52958/0 (socket says 172.21.15.23:52958) 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.330+0000 7fcffffff700 1 -- 172.21.15.23:0/1532787016 learned_addr learned my addr 172.21.15.23:0/1532787016 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fcffffff700 1 -- 172.21.15.23:0/1532787016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff000b200 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fcffffff700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd00010d550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcff000e040 tx=0x7fcff0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff001b410 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcff001b9e0 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd000070670 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff001a820 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.331+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00010dc00 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.332+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fcff000f070 con 0x7fd0000fa9d0 2023-12-08T10:06:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.332+0000 7fcffcff9700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fcfe002feb0 0x7fcfe0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:15.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.332+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcff00415e0 con 0x7fd0000fa9d0 2023-12-08T10:06:15.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.333+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd000045ad0 con 0x7fd0000fa9d0 2023-12-08T10:06:15.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.333+0000 7fcfff7fe700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fcfe002feb0 0x7fcfe0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:15.336 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.333+0000 7fcfff7fe700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fcfe002feb0 0x7fcfe0032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcfe80096a0 tx=0x7fcfe8006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:15.338 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.337+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcff0016070 con 0x7fd0000fa9d0 2023-12-08T10:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.521+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd000027a00 con 0x7fd0000fa9d0 2023-12-08T10:06:15.530 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.529+0000 7fcffcff9700 1 -- 172.21.15.23:0/1532787016 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fcff0025460 con 0x7fd0000fa9d0 2023-12-08T10:06:15.532 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.530+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fcfe002feb0 msgr2=0x7fcfe0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fcfe002feb0 0x7fcfe0032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcfe80096a0 tx=0x7fcfe8006b40).stop 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 msgr2=0x7fd00010d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd00010d550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcff000e040 tx=0x7fcff0006ee0).stop 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 shutdown_connections 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7fcfe002feb0 0x7fcfe0032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 --2- 172.21.15.23:0/1532787016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0000fa9d0 0x7fd00010d550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 >> 172.21.15.23:0/1532787016 conn(0x7fd0000f6210 msgr2=0x7fd0000f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 shutdown_connections 2023-12-08T10:06:15.533 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:15.531+0000 7fd0051ff700 1 -- 172.21.15.23:0/1532787016 wait complete. 2023-12-08T10:06:15.933 INFO:teuthology.orchestra.run.smithi023.stderr:Enabling the dashboard module... 2023-12-08T10:06:15.933 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-12-08T10:06:16.287 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f997bf7d700 1 Processor -- start 2023-12-08T10:06:16.287 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f997bf7d700 1 -- start start 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f997bf7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99740fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f997bf7d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99740fb490 con 0x7f99740fab80 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f9979d19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99740fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f9979d19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99740fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52968/0 (socket says 172.21.15.23:52968) 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.285+0000 7f9979d19700 1 -- 172.21.15.23:0/1582807131 learned_addr learned my addr 172.21.15.23:0/1582807131 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f9979d19700 1 -- 172.21.15.23:0/1582807131 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99740fb5d0 con 0x7f99740fab80 2023-12-08T10:06:16.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f9979d19700 1 --2- 172.21.15.23:0/1582807131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99740fafa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f997000b5f0 tx=0x7f9970006ee0).ready entity=mon.0 client_cookie=6e32d6fadeb64ed6 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f9978d17700 1 -- 172.21.15.23:0/1582807131 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f997000d600 con 0x7f99740fab80 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f9978d17700 1 -- 172.21.15.23:0/1582807131 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f997000dc10 con 0x7f99740fab80 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f9978d17700 1 -- 172.21.15.23:0/1582807131 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9970011cd0 con 0x7f99740fab80 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f997bf7d700 1 -- 172.21.15.23:0/1582807131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 msgr2=0x7f99740fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.286+0000 7f997bf7d700 1 --2- 172.21.15.23:0/1582807131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99740fafa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f997000b5f0 tx=0x7f9970006ee0).stop 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.287+0000 7f997bf7d700 1 -- 172.21.15.23:0/1582807131 shutdown_connections 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.287+0000 7f997bf7d700 1 --2- 172.21.15.23:0/1582807131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99740fafa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.287+0000 7f997bf7d700 1 -- 172.21.15.23:0/1582807131 >> 172.21.15.23:0/1582807131 conn(0x7f99740f63c0 msgr2=0x7f99740f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.287+0000 7f997bf7d700 1 -- 172.21.15.23:0/1582807131 shutdown_connections 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.287+0000 7f997bf7d700 1 -- 172.21.15.23:0/1582807131 wait complete. 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.287+0000 7f997bf7d700 1 Processor -- start 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f997bf7d700 1 -- start start 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f997bf7d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99741099b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:16.290 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f997bf7d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974109ea0 con 0x7f99740fab80 2023-12-08T10:06:16.291 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f9979d19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99741099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f9979d19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99741099b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52972/0 (socket says 172.21.15.23:52972) 2023-12-08T10:06:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f9979d19700 1 -- 172.21.15.23:0/3164568321 learned_addr learned my addr 172.21.15.23:0/3164568321 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f9979d19700 1 -- 172.21.15.23:0/3164568321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997000b200 con 0x7f99740fab80 2023-12-08T10:06:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.288+0000 7f9979d19700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99741099b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f997000dfb0 tx=0x7f9970006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.289+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9970011df0 con 0x7f99740fab80 2023-12-08T10:06:16.292 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.289+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997410a0a0 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.289+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f997001b8c0 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.289+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f997001a8c0 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.289+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997410a4b0 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.290+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f997000f070 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.290+0000 7f99627fc700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f995c02fe60 0x7f995c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.290+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f9970041f90 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.290+0000 7f9979518700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f995c02fe60 0x7f995c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.290+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9974059100 con 0x7f99740fab80 2023-12-08T10:06:16.293 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.291+0000 7f9979518700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f995c02fe60 0x7f995c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f99640096a0 tx=0x7f9964006b40).ready entity=mgr.14144 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:16.297 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.296+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9970010be0 con 0x7f99740fab80 2023-12-08T10:06:16.514 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:16.513+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f9974045ad0 con 0x7f99740fab80 2023-12-08T10:06:16.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:15.529137+0000 mon.smithi023 (mon.0) 84 : audit [INF] from='client.? 172.21.15.23:0/1532787016' entity='client.admin' 2023-12-08T10:06:16.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:15.784571+0000 mon.smithi023 (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:16.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:16.021796+0000 mon.smithi023 (mon.0) 86 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:16.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:16.513610+0000 mon.smithi023 (mon.0) 87 : audit [INF] from='client.? 172.21.15.23:0/3164568321' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-08T10:06:17.023 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.022+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9970024930 con 0x7f99740fab80 2023-12-08T10:06:17.024 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.023+0000 7f99627fc700 1 -- 172.21.15.23:0/3164568321 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f997002d3f0 con 0x7f99740fab80 2023-12-08T10:06:17.028 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f995c02fe60 msgr2=0x7f995c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:17.028 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f995c02fe60 0x7f995c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f99640096a0 tx=0x7f9964006b40).stop 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 msgr2=0x7f99741099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99741099b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f997000dfb0 tx=0x7f9970006ee0).stop 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 shutdown_connections 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f995c02fe60 0x7f995c032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 --2- 172.21.15.23:0/3164568321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99740fab80 0x7f99741099b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 >> 172.21.15.23:0/3164568321 conn(0x7f99740f63c0 msgr2=0x7f99740f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 shutdown_connections 2023-12-08T10:06:17.029 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.026+0000 7f997bf7d700 1 -- 172.21.15.23:0/3164568321 wait complete. 2023-12-08T10:06:17.383 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.874+0000 7faa6d363700 1 Processor -- start 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.874+0000 7faa6d363700 1 -- start start 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.874+0000 7faa6d363700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa680fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.874+0000 7faa6d363700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa680fedd0 con 0x7faa68070430 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.874+0000 7faa66ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa680fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa66ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa680fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53006/0 (socket says 172.21.15.23:53006) 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa66ffd700 1 -- 172.21.15.23:0/3975623014 learned_addr learned my addr 172.21.15.23:0/3975623014 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:17.877 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa66ffd700 1 -- 172.21.15.23:0/3975623014 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680fef10 con 0x7faa68070430 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa66ffd700 1 --2- 172.21.15.23:0/3975623014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa680fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7faa5800b5f0 tx=0x7faa58006ee0).ready entity=mon.0 client_cookie=44c8739a751ac750 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa65ffb700 1 -- 172.21.15.23:0/3975623014 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa5800d600 con 0x7faa68070430 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa65ffb700 1 -- 172.21.15.23:0/3975623014 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa5800dc10 con 0x7faa68070430 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.875+0000 7faa65ffb700 1 -- 172.21.15.23:0/3975623014 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa58011cd0 con 0x7faa68070430 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 -- 172.21.15.23:0/3975623014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 msgr2=0x7faa680fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 --2- 172.21.15.23:0/3975623014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa680fe8e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7faa5800b5f0 tx=0x7faa58006ee0).stop 2023-12-08T10:06:17.878 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 -- 172.21.15.23:0/3975623014 shutdown_connections 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 --2- 172.21.15.23:0/3975623014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa680fe8e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 -- 172.21.15.23:0/3975623014 >> 172.21.15.23:0/3975623014 conn(0x7faa680f63c0 msgr2=0x7faa680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 -- 172.21.15.23:0/3975623014 shutdown_connections 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.876+0000 7faa6d363700 1 -- 172.21.15.23:0/3975623014 wait complete. 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa6d363700 1 Processor -- start 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa6d363700 1 -- start start 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa6d363700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa6810ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa6d363700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6810aff0 con 0x7faa68070430 2023-12-08T10:06:17.879 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa66ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa6810ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa66ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa6810ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53020/0 (socket says 172.21.15.23:53020) 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.877+0000 7faa66ffd700 1 -- 172.21.15.23:0/756333499 learned_addr learned my addr 172.21.15.23:0/756333499 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa66ffd700 1 -- 172.21.15.23:0/756333499 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5800b200 con 0x7faa68070430 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa66ffd700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa6810ab00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7faa58006800 tx=0x7faa58006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa58011df0 con 0x7faa68070430 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6810b1f0 con 0x7faa68070430 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa5801b8c0 con 0x7faa68070430 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa5801a830 con 0x7faa68070430 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.878+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6810b600 con 0x7faa68070430 2023-12-08T10:06:17.880 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.879+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7faa5800f070 con 0x7faa68070430 2023-12-08T10:06:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.879+0000 7faa4f7fe700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 0x7faa54032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.879+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa68045ad0 con 0x7faa68070430 2023-12-08T10:06:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.879+0000 7faa667fc700 1 -- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 msgr2=0x7faa54032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.879+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7faa58041090 con 0x7faa68070430 2023-12-08T10:06:17.881 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.879+0000 7faa667fc700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 0x7faa54032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-08T10:06:17.888 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:17.887+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa58019300 con 0x7faa68070430 2023-12-08T10:06:18.081 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.080+0000 7faa667fc700 1 -- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 msgr2=0x7faa54032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:18.081 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.080+0000 7faa667fc700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 0x7faa54032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-08T10:06:18.085 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.084+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7faa6810c710 con 0x7faa68070430 2023-12-08T10:06:18.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.085+0000 7faa4f7fe700 1 -- 172.21.15.23:0/756333499 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7faa58016070 con 0x7faa68070430 2023-12-08T10:06:18.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:06:18.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "epoch": 13, 2023-12-08T10:06:18.086 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "available": true, 2023-12-08T10:06:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "active_name": "smithi023.tgiigg", 2023-12-08T10:06:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-08T10:06:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:06:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 msgr2=0x7faa54032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-08T10:06:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 0x7faa54032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 msgr2=0x7faa6810ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa6810ab00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7faa58006800 tx=0x7faa58006ee0).stop 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 shutdown_connections 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7faa5402feb0 0x7faa54032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 --2- 172.21.15.23:0/756333499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa68070430 0x7faa6810ab00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 >> 172.21.15.23:0/756333499 conn(0x7faa680f63c0 msgr2=0x7faa680f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 shutdown_connections 2023-12-08T10:06:18.089 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.087+0000 7faa6d363700 1 -- 172.21.15.23:0/756333499 wait complete. 2023-12-08T10:06:18.111 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:16.783602+0000 mon.smithi023 (mon.0) 88 : audit [INF] from='mgr.14144 172.21.15.23:0/2192449302' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:18.111 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:17.023904+0000 mon.smithi023 (mon.0) 89 : audit [INF] from='client.? 172.21.15.23:0/3164568321' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-08T10:06:18.111 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:17.024008+0000 mon.smithi023 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi023.tgiigg(active, since 7s) 2023-12-08T10:06:18.433 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for the mgr to restart... 2023-12-08T10:06:18.433 INFO:teuthology.orchestra.run.smithi023.stderr:Waiting for mgr epoch 13... 2023-12-08T10:06:18.433 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-08T10:06:18.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f583957b700 1 Processor -- start 2023-12-08T10:06:18.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f583957b700 1 -- start start 2023-12-08T10:06:18.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f583957b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f58340fe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:18.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f583957b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5834070770 con 0x7f5834070350 2023-12-08T10:06:18.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f5833fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f58340fe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:18.761 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f5833fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f58340fe810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51822/0 (socket says 172.21.15.23:51822) 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f5833fff700 1 -- 172.21.15.23:0/1997365268 learned_addr learned my addr 172.21.15.23:0/1997365268 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.755+0000 7f5833fff700 1 -- 172.21.15.23:0/1997365268 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340fed00 con 0x7f5834070350 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.756+0000 7f5833fff700 1 --2- 172.21.15.23:0/1997365268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f58340fe810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f582400b5f0 tx=0x7f5824006ee0).ready entity=mon.0 client_cookie=6d229d24c26cd23d server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.756+0000 7f5832ffd700 1 -- 172.21.15.23:0/1997365268 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582400d600 con 0x7f5834070350 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.756+0000 7f5832ffd700 1 -- 172.21.15.23:0/1997365268 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f582400dc10 con 0x7f5834070350 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.756+0000 7f5832ffd700 1 -- 172.21.15.23:0/1997365268 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5824011cd0 con 0x7f5834070350 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 -- 172.21.15.23:0/1997365268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 msgr2=0x7f58340fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 --2- 172.21.15.23:0/1997365268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f58340fe810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f582400b5f0 tx=0x7f5824006ee0).stop 2023-12-08T10:06:18.762 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 -- 172.21.15.23:0/1997365268 shutdown_connections 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 --2- 172.21.15.23:0/1997365268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f58340fe810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 -- 172.21.15.23:0/1997365268 >> 172.21.15.23:0/1997365268 conn(0x7f58340f6210 msgr2=0x7f58340f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 -- 172.21.15.23:0/1997365268 shutdown_connections 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.757+0000 7f583957b700 1 -- 172.21.15.23:0/1997365268 wait complete. 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.758+0000 7f583957b700 1 Processor -- start 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.758+0000 7f583957b700 1 -- start start 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.758+0000 7f583957b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f583410a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.758+0000 7f583957b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583410acf0 con 0x7f5834070350 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.758+0000 7f5833fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f583410a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5833fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f583410a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51836/0 (socket says 172.21.15.23:51836) 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5833fff700 1 -- 172.21.15.23:0/410033076 learned_addr learned my addr 172.21.15.23:0/410033076 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:18.763 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5833fff700 1 -- 172.21.15.23:0/410033076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582400b200 con 0x7f5834070350 2023-12-08T10:06:18.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5833fff700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f583410a800 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f582400b1d0 tx=0x7f5824006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:18.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5824011df0 con 0x7f5834070350 2023-12-08T10:06:18.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583410aef0 con 0x7f5834070350 2023-12-08T10:06:18.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f582401b8c0 con 0x7f5834070350 2023-12-08T10:06:18.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582401a8a0 con 0x7f5834070350 2023-12-08T10:06:18.764 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.759+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583410b300 con 0x7f5834070350 2023-12-08T10:06:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.760+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f582400f070 con 0x7f5834070350 2023-12-08T10:06:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.760+0000 7f5830ff9700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.760+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 --> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5820032a30 con 0x7f582002feb0 2023-12-08T10:06:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.760+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5824041060 con 0x7f5834070350 2023-12-08T10:06:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.760+0000 7f58337fe700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 msgr2=0x7f5820032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:18.765 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.760+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-08T10:06:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.961+0000 7f58337fe700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 msgr2=0x7f5820032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:18.962 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:18.961+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-08T10:06:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:18.085367+0000 mon.smithi023 (mon.0) 91 : audit [DBG] from='client.? 172.21.15.23:0/756333499' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-08T10:06:19.363 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:19.362+0000 7f58337fe700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 msgr2=0x7f5820032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:19.363 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:19.362+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-08T10:06:20.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:20.163+0000 7f58337fe700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 msgr2=0x7f5820032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:20.164 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:20.163+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-08T10:06:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:21.764+0000 7f58337fe700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 msgr2=0x7f5820032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3181112905 2023-12-08T10:06:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:21.764+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-08T10:06:22.302 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:22.299+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f5824024930 con 0x7f5834070350 2023-12-08T10:06:22.302 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:22.299+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 msgr2=0x7f5820032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-08T10:06:22.302 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:22.299+0000 7f5830ff9700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:06:22.554 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:22.248069+0000 mon.smithi023 (mon.0) 92 : cluster [INF] Active manager daemon smithi023.tgiigg restarted 2023-12-08T10:06:22.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:22.248589+0000 mon.smithi023 (mon.0) 93 : cluster [INF] Activating manager daemon smithi023.tgiigg 2023-12-08T10:06:22.555 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:22.255152+0000 mon.smithi023 (mon.0) 94 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-08T10:06:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.306+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f5824024410 con 0x7f5834070350 2023-12-08T10:06:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.306+0000 7f5830ff9700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5820033b40 0x7f5820035f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.306+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5820032a30 con 0x7f5820033b40 2023-12-08T10:06:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.307+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5820033b40 0x7f5820035f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.308+0000 7f58337fe700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5820033b40 0x7f5820035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5828006fd0 tx=0x7f5828006cf0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:23.310 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.309+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mgr.14164 v2:172.21.15.23:6800/2079584419 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f5820032a30 con 0x7f5820033b40 2023-12-08T10:06:23.315 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.314+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5834059100 con 0x7f5820033b40 2023-12-08T10:06:23.316 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f5830ff9700 1 -- 172.21.15.23:0/410033076 <== mgr.14164 v2:172.21.15.23:6800/2079584419 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5834059100 con 0x7f5820033b40 2023-12-08T10:06:23.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: { 2023-12-08T10:06:23.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-12-08T10:06:23.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: "initialized": true 2023-12-08T10:06:23.317 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: } 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5820033b40 msgr2=0x7f5820035f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5820033b40 0x7f5820035f30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5828006fd0 tx=0x7f5828006cf0).stop 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 msgr2=0x7f583410a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f583410a800 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f582400b1d0 tx=0x7f5824006ee0).stop 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 shutdown_connections 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5820033b40 0x7f5820035f30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834070350 0x7f583410a800 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 --2- 172.21.15.23:0/410033076 >> [v2:172.21.15.23:6800/3181112905,v1:172.21.15.23:6801/3181112905] conn(0x7f582002feb0 0x7f5820032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 >> 172.21.15.23:0/410033076 conn(0x7f58340f6210 msgr2=0x7f58340f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 shutdown_connections 2023-12-08T10:06:23.318 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:23.315+0000 7f583957b700 1 -- 172.21.15.23:0/410033076 wait complete. 2023-12-08T10:06:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:22.300984+0000 mon.smithi023 (mon.0) 95 : cluster [DBG] mgrmap e14: smithi023.tgiigg(active, starting, since 0.0525144s) 2023-12-08T10:06:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.303712+0000 mon.smithi023 (mon.0) 96 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata", "id": "smithi023"}]: dispatch 2023-12-08T10:06:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.304305+0000 mon.smithi023 (mon.0) 97 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mgr metadata", "who": "smithi023.tgiigg", "id": "smithi023.tgiigg"}]: dispatch 2023-12-08T10:06:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.305020+0000 mon.smithi023 (mon.0) 98 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-08T10:06:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.305225+0000 mon.smithi023 (mon.0) 99 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.305380+0000 mon.smithi023 (mon.0) 100 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:22.318845+0000 mon.smithi023 (mon.0) 101 : cluster [INF] Manager daemon smithi023.tgiigg is now available 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.422548+0000 mon.smithi023 (mon.0) 102 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.423677+0000 mon.smithi023 (mon.0) 103 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.441453+0000 mon.smithi023 (mon.0) 104 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:22.450522+0000 mon.smithi023 (mon.0) 105 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:06:23.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:23.237779+0000 mon.smithi023 (mon.0) 106 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:23.674 INFO:teuthology.orchestra.run.smithi023.stderr:mgr epoch 13 is available 2023-12-08T10:06:23.674 INFO:teuthology.orchestra.run.smithi023.stderr:Generating a dashboard self-signed certificate... 2023-12-08T10:06:23.674 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-12-08T10:06:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc90876700 1 Processor -- start 2023-12-08T10:06:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc90876700 1 -- start start 2023-12-08T10:06:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc90876700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c0fae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc90876700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0fb300 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c0fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:24.283 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c0fae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51872/0 (socket says 172.21.15.23:51872) 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.139+0000 7fdc8b7fe700 1 -- 172.21.15.23:0/2320110877 learned_addr learned my addr 172.21.15.23:0/2320110877 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc8b7fe700 1 -- 172.21.15.23:0/2320110877 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c0fb440 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc8b7fe700 1 --2- 172.21.15.23:0/2320110877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c0fae10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdc7c00b5f0 tx=0x7fdc7c006ee0).ready entity=mon.0 client_cookie=22e013153e89ea24 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc8a7fc700 1 -- 172.21.15.23:0/2320110877 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c00d600 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc8a7fc700 1 -- 172.21.15.23:0/2320110877 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc7c00dc10 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc8a7fc700 1 -- 172.21.15.23:0/2320110877 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c011cd0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc90876700 1 -- 172.21.15.23:0/2320110877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 msgr2=0x7fdc8c0fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc90876700 1 --2- 172.21.15.23:0/2320110877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c0fae10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdc7c00b5f0 tx=0x7fdc7c006ee0).stop 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc90876700 1 -- 172.21.15.23:0/2320110877 shutdown_connections 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc90876700 1 --2- 172.21.15.23:0/2320110877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c0fae10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.140+0000 7fdc90876700 1 -- 172.21.15.23:0/2320110877 >> 172.21.15.23:0/2320110877 conn(0x7fdc8c0f6210 msgr2=0x7fdc8c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.141+0000 7fdc90876700 1 -- 172.21.15.23:0/2320110877 shutdown_connections 2023-12-08T10:06:24.284 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.141+0000 7fdc90876700 1 -- 172.21.15.23:0/2320110877 wait complete. 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.141+0000 7fdc90876700 1 Processor -- start 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.141+0000 7fdc90876700 1 -- start start 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.141+0000 7fdc90876700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c1097e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc90876700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c109cd0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c1097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c1097e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51886/0 (socket says 172.21.15.23:51886) 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc8b7fe700 1 -- 172.21.15.23:0/3171769232 learned_addr learned my addr 172.21.15.23:0/3171769232 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc8b7fe700 1 -- 172.21.15.23:0/3171769232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c00b200 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc8b7fe700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c1097e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdc7c00dfb0 tx=0x7fdc7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:24.285 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c011c90 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc7c0228d0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c109ed0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc7c018830 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.142+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c10a2e0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.143+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fdc7c00f070 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.143+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8c045ad0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.143+0000 7fdc73fff700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdc7802fd50 0x7fdc78032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.143+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc7c042db0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.143+0000 7fdc8affd700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdc7802fd50 0x7fdc78032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.144+0000 7fdc8affd700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdc7802fd50 0x7fdc78032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdc800096a0 tx=0x7fdc80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.148+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc7c0298a0 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.313 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.311+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdc7c029d60 con 0x7fdc8c0fa9f0 2023-12-08T10:06:24.319 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.318+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc8c10abc0 con 0x7fdc7802fd50 2023-12-08T10:06:24.371 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.370+0000 7fdc73fff700 1 -- 172.21.15.23:0/3171769232 <== mgr.14164 v2:172.21.15.23:6800/2079584419 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fdc8c10abc0 con 0x7fdc7802fd50 2023-12-08T10:06:24.371 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-08T10:06:24.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.371+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdc7802fd50 msgr2=0x7fdc78032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:24.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.371+0000 7fdc90876700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdc7802fd50 0x7fdc78032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdc800096a0 tx=0x7fdc80006b40).stop 2023-12-08T10:06:24.373 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.371+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 msgr2=0x7fdc8c1097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.371+0000 7fdc90876700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c1097e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdc7c00dfb0 tx=0x7fdc7c006ee0).stop 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.372+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 shutdown_connections 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.372+0000 7fdc90876700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdc7802fd50 0x7fdc78032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.372+0000 7fdc90876700 1 --2- 172.21.15.23:0/3171769232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc8c0fa9f0 0x7fdc8c1097e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.372+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 >> 172.21.15.23:0/3171769232 conn(0x7fdc8c0f6210 msgr2=0x7fdc8c0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.372+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 shutdown_connections 2023-12-08T10:06:24.374 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:24.372+0000 7fdc90876700 1 -- 172.21.15.23:0/3171769232 wait complete. 2023-12-08T10:06:24.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:23.308602+0000 mon.smithi023 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi023.tgiigg(active, since 1.06013s) 2023-12-08T10:06:24.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:23.309095+0000 mgr.smithi023.tgiigg (mgr.14164) 1 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-08T10:06:24.396 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:23.315264+0000 mgr.smithi023.tgiigg (mgr.14164) 2 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-08T10:06:24.740 INFO:teuthology.orchestra.run.smithi023.stderr:Creating initial admin user... 2023-12-08T10:06:24.741 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_ufrp5yb:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-12-08T10:06:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.095+0000 7ff3cc459700 1 Processor -- start 2023-12-08T10:06:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.095+0000 7ff3cc459700 1 -- start start 2023-12-08T10:06:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3cc459700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3cc459700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c40fedd0 con 0x7ff3c4070430 2023-12-08T10:06:25.100 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3ca1f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3ca1f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51894/0 (socket says 172.21.15.23:51894) 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3ca1f5700 1 -- 172.21.15.23:0/1282375085 learned_addr learned my addr 172.21.15.23:0/1282375085 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3ca1f5700 1 -- 172.21.15.23:0/1282375085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c40fef10 con 0x7ff3c4070430 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3ca1f5700 1 --2- 172.21.15.23:0/1282375085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c40fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff3b800b5f0 tx=0x7ff3b8006ee0).ready entity=mon.0 client_cookie=2dd29c1084be4092 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3c91f3700 1 -- 172.21.15.23:0/1282375085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3b800d600 con 0x7ff3c4070430 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3c91f3700 1 -- 172.21.15.23:0/1282375085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3b800dc10 con 0x7ff3c4070430 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.096+0000 7ff3c91f3700 1 -- 172.21.15.23:0/1282375085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3b8011cd0 con 0x7ff3c4070430 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 -- 172.21.15.23:0/1282375085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 msgr2=0x7ff3c40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 --2- 172.21.15.23:0/1282375085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c40fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff3b800b5f0 tx=0x7ff3b8006ee0).stop 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 -- 172.21.15.23:0/1282375085 shutdown_connections 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 --2- 172.21.15.23:0/1282375085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c40fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 -- 172.21.15.23:0/1282375085 >> 172.21.15.23:0/1282375085 conn(0x7ff3c40f63c0 msgr2=0x7ff3c40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 -- 172.21.15.23:0/1282375085 shutdown_connections 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.097+0000 7ff3cc459700 1 -- 172.21.15.23:0/1282375085 wait complete. 2023-12-08T10:06:25.101 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3cc459700 1 Processor -- start 2023-12-08T10:06:25.102 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3cc459700 1 -- start start 2023-12-08T10:06:25.102 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3cc459700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c410aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:25.102 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3cc459700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c410afd0 con 0x7ff3c4070430 2023-12-08T10:06:25.102 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3ca1f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c410aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:25.102 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3ca1f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c410aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51910/0 (socket says 172.21.15.23:51910) 2023-12-08T10:06:25.102 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3ca1f5700 1 -- 172.21.15.23:0/20384257 learned_addr learned my addr 172.21.15.23:0/20384257 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:25.103 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3ca1f5700 1 -- 172.21.15.23:0/20384257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b800b200 con 0x7ff3c4070430 2023-12-08T10:06:25.103 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.098+0000 7ff3ca1f5700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c410aae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff3b800df90 tx=0x7ff3b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:25.103 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.099+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3b8011da0 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.099+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3b801b8c0 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.099+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c410b1d0 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.099+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3b801a930 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.099+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c410b5e0 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.099+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff3b800f070 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.100+0000 7ff3b6ffd700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3ac02feb0 0x7ff3ac032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.100+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3b8041160 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.100+0000 7ff3c99f4700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3ac02feb0 0x7ff3ac032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.100+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4059100 con 0x7ff3c4070430 2023-12-08T10:06:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.100+0000 7ff3c99f4700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3ac02feb0 0x7ff3ac032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff3c00096a0 tx=0x7ff3c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:25.107 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.104+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3b8016070 con 0x7ff3c4070430 2023-12-08T10:06:25.281 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.279+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff3c4045ad0 con 0x7ff3ac02feb0 2023-12-08T10:06:25.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.526+0000 7ff3b6ffd700 1 -- 172.21.15.23:0/20384257 <== mgr.14164 v2:172.21.15.23:6800/2079584419 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7ff3c4045ad0 con 0x7ff3ac02feb0 2023-12-08T10:06:25.529 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$flCRGfCeQQJewUtMbXDRl.HjTqGuXqxKy.7jS4e4LgpfrjNpj4kAO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1702029985, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3ac02feb0 msgr2=0x7ff3ac032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3ac02feb0 0x7ff3ac032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7ff3c00096a0 tx=0x7ff3c0006b40).stop 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 msgr2=0x7ff3c410aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c410aae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff3b800df90 tx=0x7ff3b8006ee0).stop 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 shutdown_connections 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3ac02feb0 0x7ff3ac032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 --2- 172.21.15.23:0/20384257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c4070430 0x7ff3c410aae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 >> 172.21.15.23:0/20384257 conn(0x7ff3c40f63c0 msgr2=0x7ff3c40f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 shutdown_connections 2023-12-08T10:06:25.531 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:25.528+0000 7ff3cc459700 1 -- 172.21.15.23:0/20384257 wait complete. 2023-12-08T10:06:25.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:24.312805+0000 mon.smithi023 (mon.0) 108 : cluster [DBG] mgrmap e16: smithi023.tgiigg(active, since 2s) 2023-12-08T10:06:25.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:24.318935+0000 mgr.smithi023.tgiigg (mgr.14164) 3 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:25.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:24.368399+0000 mon.smithi023 (mon.0) 109 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:25.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:24.370238+0000 mon.smithi023 (mon.0) 110 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:Fetching dashboard port number... 2023-12-08T10:06:25.874 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpykyhlwi5:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpd2lz4qsc:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd06140f700 1 Processor -- start 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd06140f700 1 -- start start 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd06140f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd06140f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0fb2e0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd05bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd05bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51920/0 (socket says 172.21.15.23:51920) 2023-12-08T10:06:26.451 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd05bfff700 1 -- 172.21.15.23:0/304660393 learned_addr learned my addr 172.21.15.23:0/304660393 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:26.453 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.448+0000 7fd05bfff700 1 -- 172.21.15.23:0/304660393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c0fb420 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.449+0000 7fd05bfff700 1 --2- 172.21.15.23:0/304660393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c0fadf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd04c00b5f0 tx=0x7fd04c006ee0).ready entity=mon.0 client_cookie=78023888d39435f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.449+0000 7fd05affd700 1 -- 172.21.15.23:0/304660393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd04c00d600 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.449+0000 7fd05affd700 1 -- 172.21.15.23:0/304660393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd04c00dc10 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.449+0000 7fd05affd700 1 -- 172.21.15.23:0/304660393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd04c011cd0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.449+0000 7fd06140f700 1 -- 172.21.15.23:0/304660393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 msgr2=0x7fd05c0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.449+0000 7fd06140f700 1 --2- 172.21.15.23:0/304660393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c0fadf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd04c00b5f0 tx=0x7fd04c006ee0).stop 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 -- 172.21.15.23:0/304660393 shutdown_connections 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 --2- 172.21.15.23:0/304660393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c0fadf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 -- 172.21.15.23:0/304660393 >> 172.21.15.23:0/304660393 conn(0x7fd05c0f6210 msgr2=0x7fd05c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 -- 172.21.15.23:0/304660393 shutdown_connections 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 -- 172.21.15.23:0/304660393 wait complete. 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 Processor -- start 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.450+0000 7fd06140f700 1 -- start start 2023-12-08T10:06:26.454 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd06140f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd06140f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0ff170 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd05bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd05bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c100cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51926/0 (socket says 172.21.15.23:51926) 2023-12-08T10:06:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd05bfff700 1 -- 172.21.15.23:0/3953598948 learned_addr learned my addr 172.21.15.23:0/3953598948 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:26.455 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd05bfff700 1 -- 172.21.15.23:0/3953598948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c00b200 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd05bfff700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c100cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd04c00dfb0 tx=0x7fd04c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd04c011df0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd04c01b8c0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd06140f700 1 -- 172.21.15.23:0/3953598948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c0ff310 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd04c01a8c0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.451+0000 7fd06140f700 1 -- 172.21.15.23:0/3953598948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c0ff720 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.452+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd04c00f070 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.452+0000 7fd058ff9700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd04802ff00 0x7fd0480323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.452+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd04c042250 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.452+0000 7fd05b7fe700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd04802ff00 0x7fd0480323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.452+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd034004fa0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.456 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.453+0000 7fd05b7fe700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd04802ff00 0x7fd0480323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd0500096a0 tx=0x7fd050006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:26.459 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.457+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd04c016070 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.618+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fd034005bf0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.618+0000 7fd058ff9700 1 -- 172.21.15.23:0/3953598948 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fd04c010be0 con 0x7fd05c0fa9d0 2023-12-08T10:06:26.621 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 8443 2023-12-08T10:06:26.622 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd04802ff00 msgr2=0x7fd0480323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd04802ff00 0x7fd0480323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd0500096a0 tx=0x7fd050006b40).stop 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 msgr2=0x7fd05c100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c100cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd04c00dfb0 tx=0x7fd04c006ee0).stop 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 shutdown_connections 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd04802ff00 0x7fd0480323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 --2- 172.21.15.23:0/3953598948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd05c0fa9d0 0x7fd05c100cf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 >> 172.21.15.23:0/3953598948 conn(0x7fd05c0f6210 msgr2=0x7fd05c0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 shutdown_connections 2023-12-08T10:06:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:/usr/bin/ceph: 2023-12-08T10:06:26.620+0000 7fd0427fc700 1 -- 172.21.15.23:0/3953598948 wait complete. 2023-12-08T10:06:26.805 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:26 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:25.281181+0000 mgr.smithi023.tgiigg (mgr.14164) 4 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:26.806 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:26 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:25.527305+0000 mon.smithi023 (mon.0) 111 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:26.989 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-08T10:06:26.998 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: disabled 2023-12-08T10:06:26.999 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: systemctl is-active firewalld.service 2023-12-08T10:06:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:systemctl: inactive 2023-12-08T10:06:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:firewalld.service is not enabled 2023-12-08T10:06:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-08T10:06:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:Ceph Dashboard is now available at: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: URL: https://smithi023.front.sepia.ceph.com:8443/ 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: User: admin 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: Password: pya9tthy5e 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:You can access the Ceph CLI with: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: ceph telemetry on 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:For more information see: 2023-12-08T10:06:27.011 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.011 INFO:teuthology.orchestra.run.smithi023.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-12-08T10:06:27.011 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:06:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:Bootstrap complete. 2023-12-08T10:06:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:Releasing lock 140387477441000 on /run/cephadm/40b80ed0-95b1-11ee-95a3-87774f69a715.lock 2023-12-08T10:06:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:Lock 140387477441000 released on /run/cephadm/40b80ed0-95b1-11ee-95a3-87774f69a715.lock 2023-12-08T10:06:27.045 INFO:tasks.cephadm:Fetching config... 2023-12-08T10:06:27.045 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:06:27.045 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-08T10:06:27.104 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-08T10:06:27.104 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:06:27.104 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-08T10:06:27.162 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-08T10:06:27.163 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:06:27.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/mon.smithi023/keyring of=/dev/stdout 2023-12-08T10:06:27.233 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-08T10:06:27.233 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2023-12-08T10:06:27.233 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-08T10:06:27.291 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-08T10:06:27.291 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4bDrwPrmqPrYrkAh/poL1VSh81f4XD9qxCy/Zfz0xIbScLW6fc6VumXwV88N6oDgtOYQVBBwwGS+AI0AScYGIb/SnvxjJk750DpN9mAU3UgjXpHjYp/BTmEDt7Z6Mng4jaOmmDTLALw2dgMixSi77CddOjlZ8bFhRjcz1tWoV06s0vkpcOJQ2/mtMj2FKOz9N0s0ZHh21Ydpa055WPwxu0/K/HcK4AGVVVA2suz503TGDfVnm6VzjxMqKTGUmHN1WCIRBNalSSFMNSLbev39CJ1ewx6MM7ECpcdQbOseXdnu60uc/J2MF5So/5G3TMLOuLdo0uUvnQOvOv1FUTYJgyhoxd+I0PYuegs5csXPC02swEI5ovSb48+s7RFPWaf/GVn3n/hvF/dwnCxTYUpFpVCzZFquhxuxyN5h9Y+Toy8UK64yRkR9wcFy3P9h+72QRmHt8TeI2qW0LGOz7+NlVPdZd1H1s2eOed/7l3Fxl0Jv7DU/H6rGFcM2tLFuy50= ceph-40b80ed0-95b1-11ee-95a3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-08T10:06:27.368 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4bDrwPrmqPrYrkAh/poL1VSh81f4XD9qxCy/Zfz0xIbScLW6fc6VumXwV88N6oDgtOYQVBBwwGS+AI0AScYGIb/SnvxjJk750DpN9mAU3UgjXpHjYp/BTmEDt7Z6Mng4jaOmmDTLALw2dgMixSi77CddOjlZ8bFhRjcz1tWoV06s0vkpcOJQ2/mtMj2FKOz9N0s0ZHh21Ydpa055WPwxu0/K/HcK4AGVVVA2suz503TGDfVnm6VzjxMqKTGUmHN1WCIRBNalSSFMNSLbev39CJ1ewx6MM7ECpcdQbOseXdnu60uc/J2MF5So/5G3TMLOuLdo0uUvnQOvOv1FUTYJgyhoxd+I0PYuegs5csXPC02swEI5ovSb48+s7RFPWaf/GVn3n/hvF/dwnCxTYUpFpVCzZFquhxuxyN5h9Y+Toy8UK64yRkR9wcFy3P9h+72QRmHt8TeI2qW0LGOz7+NlVPdZd1H1s2eOed/7l3Fxl0Jv7DU/H6rGFcM2tLFuy50= ceph-40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:06:27.381 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4bDrwPrmqPrYrkAh/poL1VSh81f4XD9qxCy/Zfz0xIbScLW6fc6VumXwV88N6oDgtOYQVBBwwGS+AI0AScYGIb/SnvxjJk750DpN9mAU3UgjXpHjYp/BTmEDt7Z6Mng4jaOmmDTLALw2dgMixSi77CddOjlZ8bFhRjcz1tWoV06s0vkpcOJQ2/mtMj2FKOz9N0s0ZHh21Ydpa055WPwxu0/K/HcK4AGVVVA2suz503TGDfVnm6VzjxMqKTGUmHN1WCIRBNalSSFMNSLbev39CJ1ewx6MM7ECpcdQbOseXdnu60uc/J2MF5So/5G3TMLOuLdo0uUvnQOvOv1FUTYJgyhoxd+I0PYuegs5csXPC02swEI5ovSb48+s7RFPWaf/GVn3n/hvF/dwnCxTYUpFpVCzZFquhxuxyN5h9Y+Toy8UK64yRkR9wcFy3P9h+72QRmHt8TeI2qW0LGOz7+NlVPdZd1H1s2eOed/7l3Fxl0Jv7DU/H6rGFcM2tLFuy50= ceph-40b80ed0-95b1-11ee-95a3-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-08T10:06:27.425 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCx4bDrwPrmqPrYrkAh/poL1VSh81f4XD9qxCy/Zfz0xIbScLW6fc6VumXwV88N6oDgtOYQVBBwwGS+AI0AScYGIb/SnvxjJk750DpN9mAU3UgjXpHjYp/BTmEDt7Z6Mng4jaOmmDTLALw2dgMixSi77CddOjlZ8bFhRjcz1tWoV06s0vkpcOJQ2/mtMj2FKOz9N0s0ZHh21Ydpa055WPwxu0/K/HcK4AGVVVA2suz503TGDfVnm6VzjxMqKTGUmHN1WCIRBNalSSFMNSLbev39CJ1ewx6MM7ECpcdQbOseXdnu60uc/J2MF5So/5G3TMLOuLdo0uUvnQOvOv1FUTYJgyhoxd+I0PYuegs5csXPC02swEI5ovSb48+s7RFPWaf/GVn3n/hvF/dwnCxTYUpFpVCzZFquhxuxyN5h9Y+Toy8UK64yRkR9wcFy3P9h+72QRmHt8TeI2qW0LGOz7+NlVPdZd1H1s2eOed/7l3Fxl0Jv7DU/H6rGFcM2tLFuy50= ceph-40b80ed0-95b1-11ee-95a3-87774f69a715 2023-12-08T10:06:27.438 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-08T10:06:27.755 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:26.619655+0000 mon.smithi023 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.23:0/3953598948' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-08T10:06:28.818 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 -- 172.21.15.23:0/703348566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 msgr2=0x7fefb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 --2- 172.21.15.23:0/703348566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb80f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fefa8009fb0 tx=0x7fefa8009b30).stop 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 -- 172.21.15.23:0/703348566 shutdown_connections 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 --2- 172.21.15.23:0/703348566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb80f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 -- 172.21.15.23:0/703348566 >> 172.21.15.23:0/703348566 conn(0x7fefb80f0b70 msgr2=0x7fefb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 -- 172.21.15.23:0/703348566 shutdown_connections 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.815+0000 7fefbe6f9700 1 -- 172.21.15.23:0/703348566 wait complete. 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefbe6f9700 1 Processor -- start 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefbe6f9700 1 -- start start 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefbe6f9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefbe6f9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8106c20 con 0x7fefb80f5410 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:28.819 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42300/0 (socket says 172.21.15.23:42300) 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefb7fff700 1 -- 172.21.15.23:0/631816885 learned_addr learned my addr 172.21.15.23:0/631816885 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefb7fff700 1 -- 172.21.15.23:0/631816885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8005040 con 0x7fefb80f5410 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.816+0000 7fefb7fff700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb8106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fefa8013510 tx=0x7fefa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.817+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa8009320 con 0x7fefb80f5410 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.817+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefa8013f00 con 0x7fefb80f5410 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.817+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa8014d90 con 0x7fefb80f5410 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.817+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb8106e20 con 0x7fefb80f5410 2023-12-08T10:06:28.820 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.817+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb81072b0 con 0x7fefb80f5410 2023-12-08T10:06:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.818+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb8045ad0 con 0x7fefb80f5410 2023-12-08T10:06:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.818+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefa80095f0 con 0x7fefb80f5410 2023-12-08T10:06:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.818+0000 7fefb57fa700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fefa402ff60 0x7fefa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.818+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefa8041200 con 0x7fefb80f5410 2023-12-08T10:06:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.818+0000 7fefaffff700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fefa402ff60 0x7fefa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:28.821 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.819+0000 7fefaffff700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fefa402ff60 0x7fefa4032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fefa00096a0 tx=0x7fefa0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:28.825 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.822+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefa8014420 con 0x7fefb80f5410 2023-12-08T10:06:28.989 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.987+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fefb80fae60 con 0x7fefb80f5410 2023-12-08T10:06:28.993 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.991+0000 7fefb57fa700 1 -- 172.21.15.23:0/631816885 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fefa802b030 con 0x7fefb80f5410 2023-12-08T10:06:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.993+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fefa402ff60 msgr2=0x7fefa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.993+0000 7fefbe6f9700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fefa402ff60 0x7fefa4032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fefa00096a0 tx=0x7fefa0006b40).stop 2023-12-08T10:06:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.993+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 msgr2=0x7fefb8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.993+0000 7fefbe6f9700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb8106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fefa8013510 tx=0x7fefa800b040).stop 2023-12-08T10:06:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.995+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 shutdown_connections 2023-12-08T10:06:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.995+0000 7fefbe6f9700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fefa402ff60 0x7fefa4032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.995+0000 7fefbe6f9700 1 --2- 172.21.15.23:0/631816885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefb80f5410 0x7fefb8106730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.995+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 >> 172.21.15.23:0/631816885 conn(0x7fefb80f0b70 msgr2=0x7fefb80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.995+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 shutdown_connections 2023-12-08T10:06:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:28.995+0000 7fefbe6f9700 1 -- 172.21.15.23:0/631816885 wait complete. 2023-12-08T10:06:29.372 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2023-12-08T10:06:29.372 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:06:29.372 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2023-12-08T10:06:29.394 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2023-12-08T10:06:29.394 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-08T10:06:29.455 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2023-12-08T10:06:29.456 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph orch host add smithi170 2023-12-08T10:06:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:28.785389+0000 mon.smithi023 (mon.0) 113 : cluster [DBG] mgrmap e17: smithi023.tgiigg(active, since 6s) 2023-12-08T10:06:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:28.992804+0000 mon.smithi023 (mon.0) 114 : audit [INF] from='client.? 172.21.15.23:0/631816885' entity='client.admin' 2023-12-08T10:06:31.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:30.317112+0000 mon.smithi023 (mon.0) 115 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:31.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:30.587563+0000 mon.smithi023 (mon.0) 116 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:31.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:30.594137+0000 mgr.smithi023.tgiigg (mgr.14164) 5 : cephadm [INF] Deploying daemon alertmanager.smithi023 on smithi023 2023-12-08T10:06:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:42.306425+0000 mgr.smithi023.tgiigg (mgr.14164) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:44.306725+0000 mgr.smithi023.tgiigg (mgr.14164) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:48.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:46.307094+0000 mgr.smithi023.tgiigg (mgr.14164) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:48.307462+0000 mgr.smithi023.tgiigg (mgr.14164) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 -- 172.21.15.170:0/2661861001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 msgr2=0x7f64f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 --2- 172.21.15.170:0/2661861001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f00f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f64e0009fb0 tx=0x7f64e0009b30).stop 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 -- 172.21.15.170:0/2661861001 shutdown_connections 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 --2- 172.21.15.170:0/2661861001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f00f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 -- 172.21.15.170:0/2661861001 >> 172.21.15.170:0/2661861001 conn(0x7f64f00f1c70 msgr2=0x7f64f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 -- 172.21.15.170:0/2661861001 shutdown_connections 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.417+0000 7f64f832b700 1 -- 172.21.15.170:0/2661861001 wait complete. 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f832b700 1 Processor -- start 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f832b700 1 -- start start 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f832b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:50.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f832b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0106c20 con 0x7f64f00f6510 2023-12-08T10:06:50.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f60c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:50.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f60c7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:58802/0 (socket says 172.21.15.170:58802) 2023-12-08T10:06:50.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.418+0000 7f64f60c7700 1 -- 172.21.15.170:0/2216595464 learned_addr learned my addr 172.21.15.170:0/2216595464 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:06:50.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64f60c7700 1 -- 172.21.15.170:0/2216595464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e0005040 con 0x7f64f00f6510 2023-12-08T10:06:50.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64f60c7700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f0106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f64e0014000 tx=0x7f64e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:50.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e00095f0 con 0x7f64f00f6510 2023-12-08T10:06:50.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64e0007e70 con 0x7f64f00f6510 2023-12-08T10:06:50.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f0106e20 con 0x7f64f00f6510 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64e0014ea0 con 0x7f64f00f6510 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.419+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f01072b0 con 0x7f64f00f6510 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.420+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f0045ad0 con 0x7f64f00f6510 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.420+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64e0025070 con 0x7f64f00f6510 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.420+0000 7f64e6ffd700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f64dc02ff60 0x7f64dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.420+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64e0040a30 con 0x7f64f00f6510 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.421+0000 7f64f58c6700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f64dc02ff60 0x7f64dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:50.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.421+0000 7f64f58c6700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f64dc02ff60 0x7f64dc032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f64ec0096a0 tx=0x7f64ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:50.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.426+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64e00079c0 con 0x7f64f00f6510 2023-12-08T10:06:50.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:50.598+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}) v1 -- 0x7f64f00f2e70 con 0x7f64dc02ff60 2023-12-08T10:06:51.205 INFO:teuthology.orchestra.run.smithi170.stdout:Added host 'smithi170' 2023-12-08T10:06:51.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.201+0000 7f64e6ffd700 1 -- 172.21.15.170:0/2216595464 <== mgr.14164 v2:172.21.15.23:6800/2079584419 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f64f00f2e70 con 0x7f64dc02ff60 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f64dc02ff60 msgr2=0x7f64dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f64dc02ff60 0x7f64dc032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f64ec0096a0 tx=0x7f64ec006b40).stop 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 msgr2=0x7f64f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f0106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f64e0014000 tx=0x7f64e000b040).stop 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 shutdown_connections 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f64dc02ff60 0x7f64dc032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 --2- 172.21.15.170:0/2216595464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64f00f6510 0x7f64f0106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.203+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 >> 172.21.15.170:0/2216595464 conn(0x7f64f00f1c70 msgr2=0x7f64f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.204+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 shutdown_connections 2023-12-08T10:06:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:51.204+0000 7f64f832b700 1 -- 172.21.15.170:0/2216595464 wait complete. 2023-12-08T10:06:51.551 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph orch host ls --format=json 2023-12-08T10:06:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:50.307833+0000 mgr.smithi023.tgiigg (mgr.14164) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:50.600277+0000 mgr.smithi023.tgiigg (mgr.14164) 11 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:51.201796+0000 mon.smithi023 (mon.0) 117 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 -- 172.21.15.170:0/4112379023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 msgr2=0x7f5bac0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 --2- 172.21.15.170:0/4112379023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f8f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f5b9c009fb0 tx=0x7f5b9c009b30).stop 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 -- 172.21.15.170:0/4112379023 shutdown_connections 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 --2- 172.21.15.170:0/4112379023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f8f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 -- 172.21.15.170:0/4112379023 >> 172.21.15.170:0/4112379023 conn(0x7f5bac0f09a0 msgr2=0x7f5bac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 -- 172.21.15.170:0/4112379023 shutdown_connections 2023-12-08T10:06:52.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 -- 172.21.15.170:0/4112379023 wait complete. 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.649+0000 7f5bb120f700 1 Processor -- start 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5bb120f700 1 -- start start 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5bb120f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5bb120f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac0f4e20 con 0x7f5bac0601e0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5babfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5babfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60392/0 (socket says 172.21.15.170:60392) 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5babfff700 1 -- 172.21.15.170:0/752376094 learned_addr learned my addr 172.21.15.170:0/752376094 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5babfff700 1 -- 172.21.15.170:0/752376094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c005040 con 0x7f5bac0601e0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.650+0000 7f5babfff700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f69a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5b9c013510 tx=0x7f5b9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.651+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b9c009320 con 0x7f5bac0601e0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.651+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bac0f4fc0 con 0x7f5bac0601e0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.651+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b9c013f00 con 0x7f5bac0601e0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.651+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b9c014d90 con 0x7f5bac0601e0 2023-12-08T10:06:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.651+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bac0f5450 con 0x7f5bac0601e0 2023-12-08T10:06:52.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.651+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bac059100 con 0x7f5bac0601e0 2023-12-08T10:06:52.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.652+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b9c0095f0 con 0x7f5bac0601e0 2023-12-08T10:06:52.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.652+0000 7f5ba8ff9700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b9402ff60 0x7f5b94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.652+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b9c040fe0 con 0x7f5bac0601e0 2023-12-08T10:06:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.652+0000 7f5bab7fe700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b9402ff60 0x7f5b94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:52.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.652+0000 7f5bab7fe700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b9402ff60 0x7f5b94032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5ba00096a0 tx=0x7f5ba0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:52.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.656+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b9c01d7f0 con 0x7f5bac0601e0 2023-12-08T10:06:52.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.825+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5bac0f5ca0 con 0x7f5b9402ff60 2023-12-08T10:06:52.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.827+0000 7f5ba8ff9700 1 -- 172.21.15.170:0/752376094 <== mgr.14164 v2:172.21.15.23:6800/2079584419 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f5bac0f5ca0 con 0x7f5b9402ff60 2023-12-08T10:06:52.829 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:06:52.829 INFO:teuthology.orchestra.run.smithi170.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "smithi170", "hostname": "smithi170", "labels": [], "status": ""}] 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b9402ff60 msgr2=0x7f5b94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b9402ff60 0x7f5b94032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5ba00096a0 tx=0x7f5ba0006b40).stop 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 msgr2=0x7f5bac0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f69a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f5b9c013510 tx=0x7f5b9c00b040).stop 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 shutdown_connections 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b9402ff60 0x7f5b94032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:52.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 --2- 172.21.15.170:0/752376094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bac0601e0 0x7f5bac0f69a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:52.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 >> 172.21.15.170:0/752376094 conn(0x7f5bac0f09a0 msgr2=0x7f5bac0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:52.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 shutdown_connections 2023-12-08T10:06:52.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:52.829+0000 7f5bb120f700 1 -- 172.21.15.170:0/752376094 wait complete. 2023-12-08T10:06:53.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:51.202243+0000 mgr.smithi023.tgiigg (mgr.14164) 12 : cephadm [INF] Added host smithi170 2023-12-08T10:06:53.202 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-08T10:06:53.203 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph osd crush tunables default 2023-12-08T10:06:54.169 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:52.308275+0000 mgr.smithi023.tgiigg (mgr.14164) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:54.170 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:52.827300+0000 mgr.smithi023.tgiigg (mgr.14164) 14 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-08T10:06:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 -- 172.21.15.23:0/3994268623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 msgr2=0x7fc5940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:54.356 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 --2- 172.21.15.23:0/3994268623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc5940f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc590009fb0 tx=0x7fc590009b30).stop 2023-12-08T10:06:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 -- 172.21.15.23:0/3994268623 shutdown_connections 2023-12-08T10:06:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 --2- 172.21.15.23:0/3994268623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc5940f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 -- 172.21.15.23:0/3994268623 >> 172.21.15.23:0/3994268623 conn(0x7fc5940f1c70 msgr2=0x7fc5940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 -- 172.21.15.23:0/3994268623 shutdown_connections 2023-12-08T10:06:54.357 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.354+0000 7fc59b864700 1 -- 172.21.15.23:0/3994268623 wait complete. 2023-12-08T10:06:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc59b864700 1 Processor -- start 2023-12-08T10:06:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc59b864700 1 -- start start 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc59b864700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc594106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc59b864700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc594106c20 con 0x7fc5940f6510 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc599600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc594106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc599600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc594106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46622/0 (socket says 172.21.15.23:46622) 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.355+0000 7fc599600700 1 -- 172.21.15.23:0/1702966180 learned_addr learned my addr 172.21.15.23:0/1702966180 (peer_addr_for_me v2:172.21.15.23:0/0) 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc599600700 1 -- 172.21.15.23:0/1702966180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590005040 con 0x7fc5940f6510 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc599600700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc594106730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc590014000 tx=0x7fc59000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5900095f0 con 0x7fc5940f6510 2023-12-08T10:06:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc590007e70 con 0x7fc5940f6510 2023-12-08T10:06:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc594106e20 con 0x7fc5940f6510 2023-12-08T10:06:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590014ea0 con 0x7fc5940f6510 2023-12-08T10:06:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.356+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5941072b0 con 0x7fc5940f6510 2023-12-08T10:06:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.357+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc590025070 con 0x7fc5940f6510 2023-12-08T10:06:54.360 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.357+0000 7fc589ffb700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc58002ff60 0x7fc580032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.357+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc590040a70 con 0x7fc5940f6510 2023-12-08T10:06:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.357+0000 7fc598dff700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc58002ff60 0x7fc580032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.357+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc594059100 con 0x7fc5940f6510 2023-12-08T10:06:54.361 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.358+0000 7fc598dff700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc58002ff60 0x7fc580032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc5840096a0 tx=0x7fc584006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:54.363 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.362+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc590026350 con 0x7fc5940f6510 2023-12-08T10:06:54.526 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.525+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc594045ad0 con 0x7fc5940f6510 2023-12-08T10:06:54.841 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.839+0000 7fc589ffb700 1 -- 172.21.15.23:0/1702966180 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc59002d090 con 0x7fc5940f6510 2023-12-08T10:06:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.840+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc58002ff60 msgr2=0x7fc580032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.840+0000 7fc59b864700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc58002ff60 0x7fc580032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fc5840096a0 tx=0x7fc584006b40).stop 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.840+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 msgr2=0x7fc594106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.840+0000 7fc59b864700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc594106730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc590014000 tx=0x7fc59000b040).stop 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.841+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 shutdown_connections 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.841+0000 7fc59b864700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc58002ff60 0x7fc580032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.841+0000 7fc59b864700 1 --2- 172.21.15.23:0/1702966180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5940f6510 0x7fc594106730 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.841+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 >> 172.21.15.23:0/1702966180 conn(0x7fc5940f1c70 msgr2=0x7fc5940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.841+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 shutdown_connections 2023-12-08T10:06:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2023-12-08T10:06:54.841+0000 7fc59b864700 1 -- 172.21.15.23:0/1702966180 wait complete. 2023-12-08T10:06:54.844 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2023-12-08T10:06:55.155 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:54 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:54.526478+0000 mon.smithi023 (mon.0) 118 : audit [INF] from='client.? 172.21.15.23:0/1702966180' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-08T10:06:55.248 INFO:tasks.cephadm:Adding mon.smithi023 on smithi023 2023-12-08T10:06:55.249 INFO:tasks.cephadm:Adding mon.smithi170 on smithi170 2023-12-08T10:06:55.249 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph orch apply mon '2;smithi023:172.21.15.23=smithi023;smithi170:172.21.15.170=smithi170' 2023-12-08T10:06:56.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:54.308528+0000 mgr.smithi023.tgiigg (mgr.14164) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:56.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:54.840184+0000 mon.smithi023 (mon.0) 119 : audit [INF] from='client.? 172.21.15.23:0/1702966180' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-08T10:06:56.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:54.840260+0000 mon.smithi023 (mon.0) 120 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.369+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2439916599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 msgr2=0x7f6d080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.369+0000 7f6d0e4c2700 1 --2- 172.21.15.170:0/2439916599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d080f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6cfc009fb0 tx=0x7f6cfc009b30).stop 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.369+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2439916599 shutdown_connections 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.369+0000 7f6d0e4c2700 1 --2- 172.21.15.170:0/2439916599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d080f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.369+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2439916599 >> 172.21.15.170:0/2439916599 conn(0x7f6d080f09a0 msgr2=0x7f6d080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2439916599 shutdown_connections 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2439916599 wait complete. 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0e4c2700 1 Processor -- start 2023-12-08T10:06:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0e4c2700 1 -- start start 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0e4c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d081091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0e4c2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08060240 con 0x7f6d080f51a0 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0d4c0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d081091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0d4c0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d081091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60420/0 (socket says 172.21.15.170:60420) 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.370+0000 7f6d0d4c0700 1 -- 172.21.15.170:0/2091255932 learned_addr learned my addr 172.21.15.170:0/2091255932 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6d0d4c0700 1 -- 172.21.15.170:0/2091255932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc005040 con 0x7f6d080f51a0 2023-12-08T10:06:56.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6d0d4c0700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d081091e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6cfc009830 tx=0x7f6cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:56.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cfc009320 con 0x7f6d080f51a0 2023-12-08T10:06:56.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d08060440 con 0x7f6d080f51a0 2023-12-08T10:06:56.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cfc0136c0 con 0x7f6d080f51a0 2023-12-08T10:06:56.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cfc01ee90 con 0x7f6d080f51a0 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.371+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d08109890 con 0x7f6d080f51a0 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.372+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cfc01e420 con 0x7f6d080f51a0 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.372+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d08045ad0 con 0x7f6d080f51a0 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.372+0000 7f6cf9ffb700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6cf402ff10 0x7f6cf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.372+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cfc0410b0 con 0x7f6d080f51a0 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.372+0000 7f6d0ccbf700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6cf402ff10 0x7f6cf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.373+0000 7f6d0ccbf700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6cf402ff10 0x7f6cf40323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6d040096a0 tx=0x7f6d04006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:56.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.377+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cfc0095f0 con 0x7f6d080f51a0 2023-12-08T10:06:56.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.555+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 --> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi023:172.21.15.23=smithi023;smithi170:172.21.15.170=smithi170", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d080f1ba0 con 0x7f6cf402ff10 2023-12-08T10:06:56.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.561+0000 7f6cf9ffb700 1 -- 172.21.15.170:0/2091255932 <== mgr.14164 v2:172.21.15.23:6800/2079584419 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6d080f1ba0 con 0x7f6cf402ff10 2023-12-08T10:06:56.562 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6cf402ff10 msgr2=0x7f6cf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6cf402ff10 0x7f6cf40323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6d040096a0 tx=0x7f6d04006b40).stop 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 msgr2=0x7f6d081091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d081091e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6cfc009830 tx=0x7f6cfc00b040).stop 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 shutdown_connections 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6cf402ff10 0x7f6cf40323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 --2- 172.21.15.170:0/2091255932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d080f51a0 0x7f6d081091e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 >> 172.21.15.170:0/2091255932 conn(0x7f6d080f09a0 msgr2=0x7f6d080f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 shutdown_connections 2023-12-08T10:06:56.564 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:56.562+0000 7f6d0e4c2700 1 -- 172.21.15.170:0/2091255932 wait complete. 2023-12-08T10:06:56.927 DEBUG:teuthology.orchestra.run.smithi170:mon.smithi170> sudo journalctl -f -n 0 -u ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170.service 2023-12-08T10:06:56.930 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:06:56.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:06:56.958 INFO:journalctl@ceph.mon.smithi170.smithi170.stdout:-- Logs begin at Fri 2023-12-08 09:52:33 UTC. -- 2023-12-08T10:06:57.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:56.308747+0000 mgr.smithi023.tgiigg (mgr.14164) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:06:57.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:56.556572+0000 mgr.smithi023.tgiigg (mgr.14164) 17 : audit [DBG] from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi023:172.21.15.23=smithi023;smithi170:172.21.15.170=smithi170", "target": ["mon-mgr", ""]}]: dispatch 2023-12-08T10:06:57.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:06:56.557993+0000 mgr.smithi023.tgiigg (mgr.14164) 18 : cephadm [INF] Saving service mon spec with placement smithi023:172.21.15.23=smithi023;smithi170:172.21.15.170=smithi170;count:2 2023-12-08T10:06:57.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:56.560780+0000 mon.smithi023 (mon.0) 121 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 -- 172.21.15.170:0/3938904379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 msgr2=0x7f8ccc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 --2- 172.21.15.170:0/3938904379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc0f6930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8cc0009fb0 tx=0x7f8cc0009b30).stop 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 -- 172.21.15.170:0/3938904379 shutdown_connections 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 --2- 172.21.15.170:0/3938904379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc0f6930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 -- 172.21.15.170:0/3938904379 >> 172.21.15.170:0/3938904379 conn(0x7f8ccc0f1c70 msgr2=0x7f8ccc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 -- 172.21.15.170:0/3938904379 shutdown_connections 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.013+0000 7f8cd2140700 1 -- 172.21.15.170:0/3938904379 wait complete. 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8cd2140700 1 Processor -- start 2023-12-08T10:06:58.016 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8cd2140700 1 -- start start 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8cd2140700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8cd2140700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ccc069e20 con 0x7f8ccc0f6510 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8ccb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8ccb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60440/0 (socket says 172.21.15.170:60440) 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.014+0000 7f8ccb7fe700 1 -- 172.21.15.170:0/740623511 learned_addr learned my addr 172.21.15.170:0/740623511 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8ccb7fe700 1 -- 172.21.15.170:0/740623511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0005040 con 0x7f8ccc0f6510 2023-12-08T10:06:58.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8ccb7fe700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc06b9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f8cc0013510 tx=0x7f8cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:58.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc0009320 con 0x7f8ccc0f6510 2023-12-08T10:06:58.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc0013f00 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ccc069fc0 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc0014d90 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.015+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ccc06a450 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.016+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cc0014420 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.016+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ccc059100 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.016+0000 7f8cb3fff700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8cb402ff60 0x7f8cb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.016+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cc00410c0 con 0x7f8ccc0f6510 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.016+0000 7f8ccaffd700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8cb402ff60 0x7f8cb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:06:58.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.017+0000 7f8ccaffd700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8cb402ff60 0x7f8cb4032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8cbc0096a0 tx=0x7f8cbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:06:58.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.021+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cc00095f0 con 0x7f8ccc0f6510 2023-12-08T10:06:58.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.226+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ccc0fbf60 con 0x7f8ccc0f6510 2023-12-08T10:06:58.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.226+0000 7f8cb3fff700 1 -- 172.21.15.170:0/740623511 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cc001db70 con 0x7f8ccc0f6510 2023-12-08T10:06:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:06:58.228 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:06:58.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8cb402ff60 msgr2=0x7f8cb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:58.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8cb402ff60 0x7f8cb4032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f8cbc0096a0 tx=0x7f8cbc006b40).stop 2023-12-08T10:06:58.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 msgr2=0x7f8ccc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc06b9a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f8cc0013510 tx=0x7f8cc000b040).stop 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 shutdown_connections 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8cb402ff60 0x7f8cb4032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 --2- 172.21.15.170:0/740623511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ccc0f6510 0x7f8ccc06b9a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 >> 172.21.15.170:0/740623511 conn(0x7f8ccc0f1c70 msgr2=0x7f8ccc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 shutdown_connections 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:06:58.228+0000 7f8cd2140700 1 -- 172.21.15.170:0/740623511 wait complete. 2023-12-08T10:06:58.230 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:06:59.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:06:58.227026+0000 mon.smithi023 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.170:0/740623511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:06:59.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:06:59.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:00.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:06:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:06:58.309098+0000 mgr.smithi023.tgiigg (mgr.14164) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/865049117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 msgr2=0x7f9dd00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 --2- 172.21.15.170:0/865049117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd00f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9dcc009fb0 tx=0x7f9dcc009b30).stop 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/865049117 shutdown_connections 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 --2- 172.21.15.170:0/865049117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd00f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/865049117 >> 172.21.15.170:0/865049117 conn(0x7f9dd00f09a0 msgr2=0x7f9dd00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/865049117 shutdown_connections 2023-12-08T10:07:00.694 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/865049117 wait complete. 2023-12-08T10:07:00.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.692+0000 7f9dd6d2a700 1 Processor -- start 2023-12-08T10:07:00.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.693+0000 7f9dd6d2a700 1 -- start start 2023-12-08T10:07:00.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.693+0000 7f9dd6d2a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:00.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.693+0000 7f9dd6d2a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd01058a0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.693+0000 7f9dd5d28700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:00.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.693+0000 7f9dd5d28700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60454/0 (socket says 172.21.15.170:60454) 2023-12-08T10:07:00.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.693+0000 7f9dd5d28700 1 -- 172.21.15.170:0/1022367028 learned_addr learned my addr 172.21.15.170:0/1022367028 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:00.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dd5d28700 1 -- 172.21.15.170:0/1022367028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc005040 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dd5d28700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd01053b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9dcc013510 tx=0x7f9dcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dcc009320 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dcc013f00 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd0105aa0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dcc014dc0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.694+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd0105f30 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.695+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dcc0095f0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.695+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd0045ad0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.695+0000 7f9dc67fc700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9dbc02ff60 0x7f9dbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.695+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dcc0410c0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.695+0000 7f9dd5527700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9dbc02ff60 0x7f9dbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:00.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.696+0000 7f9dd5527700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9dbc02ff60 0x7f9dbc032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9dc00096a0 tx=0x7f9dc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:00.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.700+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dcc01c790 con 0x7f9dd00f51a0 2023-12-08T10:07:00.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.903+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dd0106de0 con 0x7f9dd00f51a0 2023-12-08T10:07:00.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.903+0000 7f9dc67fc700 1 -- 172.21.15.170:0/1022367028 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9dcc010230 con 0x7f9dd00f51a0 2023-12-08T10:07:00.905 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:00.905 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:00.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9dbc02ff60 msgr2=0x7f9dbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:00.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9dbc02ff60 0x7f9dbc032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f9dc00096a0 tx=0x7f9dc0006b40).stop 2023-12-08T10:07:00.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 msgr2=0x7f9dd01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:00.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd01053b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f9dcc013510 tx=0x7f9dcc00b040).stop 2023-12-08T10:07:00.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 shutdown_connections 2023-12-08T10:07:00.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9dbc02ff60 0x7f9dbc032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:00.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 --2- 172.21.15.170:0/1022367028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9dd00f51a0 0x7f9dd01053b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:00.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 >> 172.21.15.170:0/1022367028 conn(0x7f9dd00f09a0 msgr2=0x7f9dd00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:00.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 shutdown_connections 2023-12-08T10:07:00.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:00.905+0000 7f9dd6d2a700 1 -- 172.21.15.170:0/1022367028 wait complete. 2023-12-08T10:07:00.909 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:02.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:00.309435+0000 mgr.smithi023.tgiigg (mgr.14164) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:02.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:00.904069+0000 mon.smithi023 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.170:0/1022367028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:02.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:02.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 -- 172.21.15.170:0/310405952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 msgr2=0x7fd7680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 --2- 172.21.15.170:0/310405952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd7680f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd758009fb0 tx=0x7fd758009b30).stop 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 -- 172.21.15.170:0/310405952 shutdown_connections 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 --2- 172.21.15.170:0/310405952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd7680f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 -- 172.21.15.170:0/310405952 >> 172.21.15.170:0/310405952 conn(0x7fd7680f1c70 msgr2=0x7fd7680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 -- 172.21.15.170:0/310405952 shutdown_connections 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.728+0000 7fd76d9ad700 1 -- 172.21.15.170:0/310405952 wait complete. 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd76d9ad700 1 Processor -- start 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd76d9ad700 1 -- start start 2023-12-08T10:07:03.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd76d9ad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd76806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd76d9ad700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd768069e20 con 0x7fd7680f6510 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd766ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd76806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd766ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd76806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34658/0 (socket says 172.21.15.170:34658) 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.729+0000 7fd766ffd700 1 -- 172.21.15.170:0/1752696269 learned_addr learned my addr 172.21.15.170:0/1752696269 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd766ffd700 1 -- 172.21.15.170:0/1752696269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758005040 con 0x7fd7680f6510 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd766ffd700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd76806b9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd758014000 tx=0x7fd75800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7580095f0 con 0x7fd7680f6510 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd758007e70 con 0x7fd7680f6510 2023-12-08T10:07:03.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd768069fc0 con 0x7fd7680f6510 2023-12-08T10:07:03.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd758014ea0 con 0x7fd7680f6510 2023-12-08T10:07:03.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.730+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76806a450 con 0x7fd7680f6510 2023-12-08T10:07:03.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.731+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd758025070 con 0x7fd7680f6510 2023-12-08T10:07:03.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.731+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd768059100 con 0x7fd7680f6510 2023-12-08T10:07:03.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.731+0000 7fd75f7fe700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd75402ff60 0x7fd754032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:03.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.731+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd758040970 con 0x7fd7680f6510 2023-12-08T10:07:03.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.732+0000 7fd7667fc700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd75402ff60 0x7fd754032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:03.735 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.732+0000 7fd7667fc700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd75402ff60 0x7fd754032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd7500096a0 tx=0x7fd750006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:03.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.736+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7580079c0 con 0x7fd7680f6510 2023-12-08T10:07:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.944+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7680fbf60 con 0x7fd7680f6510 2023-12-08T10:07:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.944+0000 7fd75f7fe700 1 -- 172.21.15.170:0/1752696269 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd75800f300 con 0x7fd7680f6510 2023-12-08T10:07:03.946 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:03.946 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd75402ff60 msgr2=0x7fd754032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd75402ff60 0x7fd754032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fd7500096a0 tx=0x7fd750006b40).stop 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 msgr2=0x7fd76806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd76806b9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd758014000 tx=0x7fd75800b040).stop 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 shutdown_connections 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd75402ff60 0x7fd754032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 --2- 172.21.15.170:0/1752696269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd7680f6510 0x7fd76806b9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 >> 172.21.15.170:0/1752696269 conn(0x7fd7680f1c70 msgr2=0x7fd7680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 shutdown_connections 2023-12-08T10:07:03.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:03.946+0000 7fd76d9ad700 1 -- 172.21.15.170:0/1752696269 wait complete. 2023-12-08T10:07:03.949 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:04.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:02.309785+0000 mgr.smithi023.tgiigg (mgr.14164) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:05.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:03.945016+0000 mon.smithi023 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.170:0/1752696269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:05.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:05.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:06.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:04.310213+0000 mgr.smithi023.tgiigg (mgr.14164) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:06.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.481+0000 7feae5e70700 1 -- 172.21.15.170:0/1323060795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 msgr2=0x7feae00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:06.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.481+0000 7feae5e70700 1 --2- 172.21.15.170:0/1323060795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae00f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fead0009fb0 tx=0x7fead0009b30).stop 2023-12-08T10:07:06.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.481+0000 7feae5e70700 1 -- 172.21.15.170:0/1323060795 shutdown_connections 2023-12-08T10:07:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.481+0000 7feae5e70700 1 --2- 172.21.15.170:0/1323060795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae00f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.481+0000 7feae5e70700 1 -- 172.21.15.170:0/1323060795 >> 172.21.15.170:0/1323060795 conn(0x7feae00f0b70 msgr2=0x7feae00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.482+0000 7feae5e70700 1 -- 172.21.15.170:0/1323060795 shutdown_connections 2023-12-08T10:07:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.482+0000 7feae5e70700 1 -- 172.21.15.170:0/1323060795 wait complete. 2023-12-08T10:07:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.482+0000 7feae5e70700 1 Processor -- start 2023-12-08T10:07:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.482+0000 7feae5e70700 1 -- start start 2023-12-08T10:07:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.483+0000 7feae5e70700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.483+0000 7feae5e70700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0060240 con 0x7feae00f5410 2023-12-08T10:07:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.483+0000 7feadf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.483+0000 7feadf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34672/0 (socket says 172.21.15.170:34672) 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.483+0000 7feadf7fe700 1 -- 172.21.15.170:0/4199429992 learned_addr learned my addr 172.21.15.170:0/4199429992 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.483+0000 7feadf7fe700 1 -- 172.21.15.170:0/4199429992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0005040 con 0x7feae00f5410 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.484+0000 7feadf7fe700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae0109430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fead0013510 tx=0x7fead000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.484+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead0009320 con 0x7feae00f5410 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.484+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae0060440 con 0x7feae00f5410 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.484+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fead0013f00 con 0x7feae00f5410 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.484+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead0014d90 con 0x7feae00f5410 2023-12-08T10:07:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.484+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae0109ae0 con 0x7feae00f5410 2023-12-08T10:07:06.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.485+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fead00095f0 con 0x7feae00f5410 2023-12-08T10:07:06.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.485+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae0045ad0 con 0x7feae00f5410 2023-12-08T10:07:06.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.485+0000 7feac7fff700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7feac802ff60 0x7feac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:06.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.485+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fead0040fc0 con 0x7feae00f5410 2023-12-08T10:07:06.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.485+0000 7feadeffd700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7feac802ff60 0x7feac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:06.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.486+0000 7feadeffd700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7feac802ff60 0x7feac8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fead40096a0 tx=0x7fead4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:06.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.490+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fead0014420 con 0x7feae00f5410 2023-12-08T10:07:06.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.696+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feae00fae60 con 0x7feae00f5410 2023-12-08T10:07:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.696+0000 7feac7fff700 1 -- 172.21.15.170:0/4199429992 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fead0010230 con 0x7feae00f5410 2023-12-08T10:07:06.698 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:06.698 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7feac802ff60 msgr2=0x7feac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7feac802ff60 0x7feac8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fead40096a0 tx=0x7fead4006b40).stop 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 msgr2=0x7feae0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae0109430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fead0013510 tx=0x7fead000b040).stop 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 shutdown_connections 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7feac802ff60 0x7feac8032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 --2- 172.21.15.170:0/4199429992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae00f5410 0x7feae0109430 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 >> 172.21.15.170:0/4199429992 conn(0x7feae00f0b70 msgr2=0x7feae00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 shutdown_connections 2023-12-08T10:07:06.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:06.698+0000 7feae5e70700 1 -- 172.21.15.170:0/4199429992 wait complete. 2023-12-08T10:07:06.701 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:07.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:06.697102+0000 mon.smithi023 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.170:0/4199429992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:08.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:08.086 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:08.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:06.310546+0000 mgr.smithi023.tgiigg (mgr.14164) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 -- 172.21.15.170:0/1186168815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 msgr2=0x7f93a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 --2- 172.21.15.170:0/1186168815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a00f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9394009fb0 tx=0x7f9394009b30).stop 2023-12-08T10:07:09.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 -- 172.21.15.170:0/1186168815 shutdown_connections 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 --2- 172.21.15.170:0/1186168815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a00f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 -- 172.21.15.170:0/1186168815 >> 172.21.15.170:0/1186168815 conn(0x7f93a00f1c70 msgr2=0x7f93a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 -- 172.21.15.170:0/1186168815 shutdown_connections 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.225+0000 7f93a5651700 1 -- 172.21.15.170:0/1186168815 wait complete. 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f93a5651700 1 Processor -- start 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f93a5651700 1 -- start start 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f93a5651700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f93a5651700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0106c20 con 0x7f93a00f6510 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f939effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f939effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34700/0 (socket says 172.21.15.170:34700) 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.226+0000 7f939effd700 1 -- 172.21.15.170:0/2974262921 learned_addr learned my addr 172.21.15.170:0/2974262921 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f939effd700 1 -- 172.21.15.170:0/2974262921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394005040 con 0x7f93a00f6510 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f939effd700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a0106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9394013510 tx=0x7f939400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394009320 con 0x7f93a00f6510 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a0106e20 con 0x7f93a00f6510 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9394013f00 con 0x7f93a00f6510 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394014d90 con 0x7f93a00f6510 2023-12-08T10:07:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.227+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a01072b0 con 0x7f93a00f6510 2023-12-08T10:07:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.228+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9394014420 con 0x7f93a00f6510 2023-12-08T10:07:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.228+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a0045ad0 con 0x7f93a00f6510 2023-12-08T10:07:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.228+0000 7f93877fe700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f938802ff60 0x7f9388032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.228+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93940410c0 con 0x7f93a00f6510 2023-12-08T10:07:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.228+0000 7f939e7fc700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f938802ff60 0x7f9388032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.229+0000 7f939e7fc700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f938802ff60 0x7f9388032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f93900096a0 tx=0x7f9390006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:09.234 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.233+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93940095f0 con 0x7f93a00f6510 2023-12-08T10:07:09.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.439+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93a00fbf60 con 0x7f93a00f6510 2023-12-08T10:07:09.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.440+0000 7f93877fe700 1 -- 172.21.15.170:0/2974262921 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f939401db70 con 0x7f93a00f6510 2023-12-08T10:07:09.441 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:09.441 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f938802ff60 msgr2=0x7f9388032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f938802ff60 0x7f9388032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f93900096a0 tx=0x7f9390006b40).stop 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 msgr2=0x7f93a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a0106730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9394013510 tx=0x7f939400b040).stop 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 shutdown_connections 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f938802ff60 0x7f9388032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 --2- 172.21.15.170:0/2974262921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93a00f6510 0x7f93a0106730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.441+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 >> 172.21.15.170:0/2974262921 conn(0x7f93a00f1c70 msgr2=0x7f93a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.442+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 shutdown_connections 2023-12-08T10:07:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:09.442+0000 7f93a5651700 1 -- 172.21.15.170:0/2974262921 wait complete. 2023-12-08T10:07:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:10.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:08.310943+0000 mgr.smithi023.tgiigg (mgr.14164) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:10.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:09.440451+0000 mon.smithi023 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.170:0/2974262921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:10.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:10.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:12.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:10.311222+0000 mgr.smithi023.tgiigg (mgr.14164) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:12.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- 172.21.15.170:0/2519457467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 msgr2=0x7f572c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 --2- 172.21.15.170:0/2519457467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c0f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f571c009fb0 tx=0x7f571c009b30).stop 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- 172.21.15.170:0/2519457467 shutdown_connections 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 --2- 172.21.15.170:0/2519457467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c0f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- 172.21.15.170:0/2519457467 >> 172.21.15.170:0/2519457467 conn(0x7f572c0f1c70 msgr2=0x7f572c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- 172.21.15.170:0/2519457467 shutdown_connections 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- 172.21.15.170:0/2519457467 wait complete. 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 Processor -- start 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- start start 2023-12-08T10:07:12.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.765+0000 7f573443a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c106c20 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.766+0000 7f57321d6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.766+0000 7f57321d6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60534/0 (socket says 172.21.15.170:60534) 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.766+0000 7f57321d6700 1 -- 172.21.15.170:0/3431693152 learned_addr learned my addr 172.21.15.170:0/3431693152 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.766+0000 7f57321d6700 1 -- 172.21.15.170:0/3431693152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c005040 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.766+0000 7f57321d6700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f571c013510 tx=0x7f571c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.767+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f571c009320 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.767+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f571c013f00 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.767+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c106e20 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.767+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f571c014d90 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.767+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c1072b0 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.767+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f571c014420 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.768+0000 7f5722ffd700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f571802ff60 0x7f5718032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.768+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f571c041070 con 0x7f572c0f6510 2023-12-08T10:07:12.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.768+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c045ad0 con 0x7f572c0f6510 2023-12-08T10:07:12.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.768+0000 7f57319d5700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f571802ff60 0x7f5718032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:12.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.768+0000 7f57319d5700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f571802ff60 0x7f5718032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:12.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.772+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f571c0095f0 con 0x7f572c0f6510 2023-12-08T10:07:12.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.983+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f572c0fc050 con 0x7f572c0f6510 2023-12-08T10:07:12.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.984+0000 7f5722ffd700 1 -- 172.21.15.170:0/3431693152 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f571c01db70 con 0x7f572c0f6510 2023-12-08T10:07:12.985 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:12.985 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:12.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f571802ff60 msgr2=0x7f5718032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f571802ff60 0x7f5718032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).stop 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 msgr2=0x7f572c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f571c013510 tx=0x7f571c00b040).stop 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 shutdown_connections 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f571802ff60 0x7f5718032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 --2- 172.21.15.170:0/3431693152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f572c0f6510 0x7f572c106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 >> 172.21.15.170:0/3431693152 conn(0x7f572c0f1c70 msgr2=0x7f572c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 shutdown_connections 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:12.986+0000 7f573443a700 1 -- 172.21.15.170:0/3431693152 wait complete. 2023-12-08T10:07:12.988 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:12.311427+0000 mgr.smithi023.tgiigg (mgr.14164) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:12.984510+0000 mon.smithi023 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.170:0/3431693152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:14.311774+0000 mgr.smithi023.tgiigg (mgr.14164) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:16.238 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:16.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:16.312162+0000 mgr.smithi023.tgiigg (mgr.14164) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 -- 172.21.15.170:0/2696572626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 msgr2=0x7f13280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 --2- 172.21.15.170:0/2696572626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f13280f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f1318009fb0 tx=0x7f1318009b30).stop 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 -- 172.21.15.170:0/2696572626 shutdown_connections 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 --2- 172.21.15.170:0/2696572626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f13280f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 -- 172.21.15.170:0/2696572626 >> 172.21.15.170:0/2696572626 conn(0x7f13280f0b70 msgr2=0x7f13280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 -- 172.21.15.170:0/2696572626 shutdown_connections 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 -- 172.21.15.170:0/2696572626 wait complete. 2023-12-08T10:07:18.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 Processor -- start 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.185+0000 7f13306cf700 1 -- start start 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f13306cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f1328106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f13306cf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1328106c00 con 0x7f13280f5410 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f132e46b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f1328106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f132e46b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f1328106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60556/0 (socket says 172.21.15.170:60556) 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f132e46b700 1 -- 172.21.15.170:0/1340504118 learned_addr learned my addr 172.21.15.170:0/1340504118 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f132e46b700 1 -- 172.21.15.170:0/1340504118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318005040 con 0x7f13280f5410 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f132e46b700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f1328106710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1318013510 tx=0x7f131800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1318009320 con 0x7f13280f5410 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1318013f00 con 0x7f13280f5410 2023-12-08T10:07:18.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1328106e00 con 0x7f13280f5410 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.186+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1318014d90 con 0x7f13280f5410 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.187+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1328107290 con 0x7f13280f5410 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.187+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13180263a0 con 0x7f13280f5410 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.187+0000 7f131effd700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f131402ff60 0x7f1314032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.187+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13180410d0 con 0x7f13280f5410 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.187+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1328045ad0 con 0x7f13280f5410 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.188+0000 7f132dc6a700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f131402ff60 0x7f1314032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:18.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.188+0000 7f132dc6a700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f131402ff60 0x7f1314032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f13240096a0 tx=0x7f1324006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:18.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.192+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1318012920 con 0x7f13280f5410 2023-12-08T10:07:18.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.396+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13280fae60 con 0x7f13280f5410 2023-12-08T10:07:18.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.397+0000 7f131effd700 1 -- 172.21.15.170:0/1340504118 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f131802b020 con 0x7f13280f5410 2023-12-08T10:07:18.400 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:18.400 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:18.401 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f131402ff60 msgr2=0x7f1314032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:18.401 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f131402ff60 0x7f1314032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f13240096a0 tx=0x7f1324006b40).stop 2023-12-08T10:07:18.401 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 msgr2=0x7f1328106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f1328106710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f1318013510 tx=0x7f131800b040).stop 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 shutdown_connections 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f131402ff60 0x7f1314032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 --2- 172.21.15.170:0/1340504118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13280f5410 0x7f1328106710 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 >> 172.21.15.170:0/1340504118 conn(0x7f13280f0b70 msgr2=0x7f132805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 shutdown_connections 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:18.399+0000 7f13306cf700 1 -- 172.21.15.170:0/1340504118 wait complete. 2023-12-08T10:07:18.402 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:18.398700+0000 mon.smithi023 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.170:0/1340504118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:19.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:19.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:20.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:18.312416+0000 mgr.smithi023.tgiigg (mgr.14164) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:20.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 -- 172.21.15.170:0/2941192923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 msgr2=0x7f52040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 --2- 172.21.15.170:0/2941192923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f52040f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f51fc009fb0 tx=0x7f51fc009b30).stop 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 -- 172.21.15.170:0/2941192923 shutdown_connections 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 --2- 172.21.15.170:0/2941192923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f52040f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 -- 172.21.15.170:0/2941192923 >> 172.21.15.170:0/2941192923 conn(0x7f52040f1c70 msgr2=0x7f52040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 -- 172.21.15.170:0/2941192923 shutdown_connections 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.913+0000 7f5208cd5700 1 -- 172.21.15.170:0/2941192923 wait complete. 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f5208cd5700 1 Processor -- start 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f5208cd5700 1 -- start start 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f5208cd5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f5204106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f5208cd5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5204106c20 con 0x7f52040f6510 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f520259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f5204106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f520259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f5204106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60572/0 (socket says 172.21.15.170:60572) 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f520259c700 1 -- 172.21.15.170:0/2526136505 learned_addr learned my addr 172.21.15.170:0/2526136505 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f520259c700 1 -- 172.21.15.170:0/2526136505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc005040 con 0x7f52040f6510 2023-12-08T10:07:20.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.914+0000 7f520259c700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f5204106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f51fc013510 tx=0x7f51fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51fc009320 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51fc013f00 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5204106e20 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51fc014d90 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52041072b0 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51fc014420 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.915+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5204045ad0 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.916+0000 7f51faffd700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f51f002ff60 0x7f51f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.916+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51fc0410c0 con 0x7f52040f6510 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.916+0000 7f5201d9b700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f51f002ff60 0x7f51f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:20.919 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.916+0000 7f5201d9b700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f51f002ff60 0x7f51f0032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f51ec0096a0 tx=0x7f51ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:20.922 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:20.920+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51fc0095f0 con 0x7f52040f6510 2023-12-08T10:07:21.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.126+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52040fbf60 con 0x7f52040f6510 2023-12-08T10:07:21.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.127+0000 7f51faffd700 1 -- 172.21.15.170:0/2526136505 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f51fc01db70 con 0x7f52040f6510 2023-12-08T10:07:21.129 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:21.129 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f51f002ff60 msgr2=0x7f51f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f51f002ff60 0x7f51f0032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f51ec0096a0 tx=0x7f51ec006b40).stop 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 msgr2=0x7f5204106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f5204106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f51fc013510 tx=0x7f51fc00b040).stop 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 shutdown_connections 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f51f002ff60 0x7f51f0032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 --2- 172.21.15.170:0/2526136505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52040f6510 0x7f5204106730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:21.131 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 >> 172.21.15.170:0/2526136505 conn(0x7f52040f1c70 msgr2=0x7f52040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:21.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 shutdown_connections 2023-12-08T10:07:21.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:21.129+0000 7f5208cd5700 1 -- 172.21.15.170:0/2526136505 wait complete. 2023-12-08T10:07:21.132 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:20.312658+0000 mgr.smithi023.tgiigg (mgr.14164) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:21.128627+0000 mon.smithi023 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.170:0/2526136505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:22.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:22.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:22.449140+0000 mon.smithi023 (mon.0) 130 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:07:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:22.452272+0000 mon.smithi023 (mon.0) 131 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:07:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:22.312937+0000 mgr.smithi023.tgiigg (mgr.14164) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:26.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:24.313308+0000 mgr.smithi023.tgiigg (mgr.14164) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:26.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 -- 172.21.15.170:0/1400443368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 msgr2=0x7f9abc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:26.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 --2- 172.21.15.170:0/1400443368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc0f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9aac009fb0 tx=0x7f9aac009b30).stop 2023-12-08T10:07:26.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 -- 172.21.15.170:0/1400443368 shutdown_connections 2023-12-08T10:07:26.953 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 --2- 172.21.15.170:0/1400443368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc0f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 -- 172.21.15.170:0/1400443368 >> 172.21.15.170:0/1400443368 conn(0x7f9abc0f1c70 msgr2=0x7f9abc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 -- 172.21.15.170:0/1400443368 shutdown_connections 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.949+0000 7f9ac4775700 1 -- 172.21.15.170:0/1400443368 wait complete. 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac4775700 1 Processor -- start 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac4775700 1 -- start start 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac4775700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac4775700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc069e20 con 0x7f9abc0f6510 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac2511700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac2511700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54242/0 (socket says 172.21.15.170:54242) 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.950+0000 7f9ac2511700 1 -- 172.21.15.170:0/2156718328 learned_addr learned my addr 172.21.15.170:0/2156718328 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ac2511700 1 -- 172.21.15.170:0/2156718328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac005040 con 0x7f9abc0f6510 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ac2511700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc06b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9aac013510 tx=0x7f9aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aac009320 con 0x7f9abc0f6510 2023-12-08T10:07:26.954 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc069fc0 con 0x7f9abc0f6510 2023-12-08T10:07:26.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9aac013f00 con 0x7f9abc0f6510 2023-12-08T10:07:26.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aac014d90 con 0x7f9abc0f6510 2023-12-08T10:07:26.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.951+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc06a450 con 0x7f9abc0f6510 2023-12-08T10:07:26.955 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.952+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9aac0263a0 con 0x7f9abc0f6510 2023-12-08T10:07:26.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.952+0000 7f9ab2ffd700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9aa802ff60 0x7f9aa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:26.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.952+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9aac041320 con 0x7f9abc0f6510 2023-12-08T10:07:26.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.952+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc059100 con 0x7f9abc0f6510 2023-12-08T10:07:26.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.952+0000 7f9ac1d10700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9aa802ff60 0x7f9aa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:26.956 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.953+0000 7f9ac1d10700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9aa802ff60 0x7f9aa8032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9ab80096a0 tx=0x7f9ab8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:26.960 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:26.957+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9aac012920 con 0x7f9abc0f6510 2023-12-08T10:07:27.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.160+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9abc06ab50 con 0x7f9abc0f6510 2023-12-08T10:07:27.163 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.161+0000 7f9ab2ffd700 1 -- 172.21.15.170:0/2156718328 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9aac02b030 con 0x7f9abc0f6510 2023-12-08T10:07:27.163 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:27.163 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.162+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9aa802ff60 msgr2=0x7f9aa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.162+0000 7f9ac4775700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9aa802ff60 0x7f9aa8032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9ab80096a0 tx=0x7f9ab8006b40).stop 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 msgr2=0x7f9abc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc06b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9aac013510 tx=0x7f9aac00b040).stop 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 shutdown_connections 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9aa802ff60 0x7f9aa8032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 --2- 172.21.15.170:0/2156718328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9abc0f6510 0x7f9abc06b9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 >> 172.21.15.170:0/2156718328 conn(0x7f9abc0f1c70 msgr2=0x7f9abc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 shutdown_connections 2023-12-08T10:07:27.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:27.163+0000 7f9ac4775700 1 -- 172.21.15.170:0/2156718328 wait complete. 2023-12-08T10:07:27.166 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:28.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:26.313611+0000 mgr.smithi023.tgiigg (mgr.14164) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:28.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:27.162747+0000 mon.smithi023 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.170:0/2156718328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:28.563 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:28.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.678+0000 7f8f517a9700 1 -- 172.21.15.170:0/1840493468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 msgr2=0x7f8f4c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 --2- 172.21.15.170:0/1840493468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0f55c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8f3c009fb0 tx=0x7f8f3c009b30).stop 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 -- 172.21.15.170:0/1840493468 shutdown_connections 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 --2- 172.21.15.170:0/1840493468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0f55c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 -- 172.21.15.170:0/1840493468 >> 172.21.15.170:0/1840493468 conn(0x7f8f4c0f09a0 msgr2=0x7f8f4c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 -- 172.21.15.170:0/1840493468 shutdown_connections 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 -- 172.21.15.170:0/1840493468 wait complete. 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 Processor -- start 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 -- start start 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:29.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.679+0000 7f8f517a9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4c0fb5a0 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f4bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f4bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54258/0 (socket says 172.21.15.170:54258) 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f4bfff700 1 -- 172.21.15.170:0/3429009177 learned_addr learned my addr 172.21.15.170:0/3429009177 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f4bfff700 1 -- 172.21.15.170:0/3429009177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c005040 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f4bfff700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0fd120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8f3c009830 tx=0x7f8f3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3c009320 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4c0fb740 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f3c0136c0 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f3c01ee90 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.680+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4c0fbbd0 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.681+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f3c01e420 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.681+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4c059100 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.681+0000 7f8f48ff9700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8f3402ff10 0x7f8f340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:29.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.681+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f3c040f60 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.682+0000 7f8f4b7fe700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8f3402ff10 0x7f8f340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:29.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.682+0000 7f8f4b7fe700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8f3402ff10 0x7f8f340323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8f400096a0 tx=0x7f8f40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:29.688 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.686+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f3c01c790 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.898+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f4c045ad0 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.899+0000 7f8f48ff9700 1 -- 172.21.15.170:0/3429009177 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f3c02c2d0 con 0x7f8f4c0f51a0 2023-12-08T10:07:29.901 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:29.902 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8f3402ff10 msgr2=0x7f8f340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8f3402ff10 0x7f8f340323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f8f400096a0 tx=0x7f8f40006b40).stop 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 msgr2=0x7f8f4c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0fd120 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8f3c009830 tx=0x7f8f3c00b040).stop 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 shutdown_connections 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8f3402ff10 0x7f8f340323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:29.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 --2- 172.21.15.170:0/3429009177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f4c0f51a0 0x7f8f4c0fd120 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:29.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 >> 172.21.15.170:0/3429009177 conn(0x7f8f4c0f09a0 msgr2=0x7f8f4c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:29.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 shutdown_connections 2023-12-08T10:07:29.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:29.901+0000 7f8f517a9700 1 -- 172.21.15.170:0/3429009177 wait complete. 2023-12-08T10:07:29.904 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:28.313959+0000 mgr.smithi023.tgiigg (mgr.14164) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:31.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:29.900524+0000 mon.smithi023 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.170:0/3429009177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:31.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:31.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:30.314295+0000 mgr.smithi023.tgiigg (mgr.14164) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- 172.21.15.170:0/1775452557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 msgr2=0x7f88540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 --2- 172.21.15.170:0/1775452557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88540f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8844009fb0 tx=0x7f8844009b30).stop 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- 172.21.15.170:0/1775452557 shutdown_connections 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 --2- 172.21.15.170:0/1775452557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88540f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- 172.21.15.170:0/1775452557 >> 172.21.15.170:0/1775452557 conn(0x7f88540f09a0 msgr2=0x7f88540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- 172.21.15.170:0/1775452557 shutdown_connections 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- 172.21.15.170:0/1775452557 wait complete. 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 Processor -- start 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- start start 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88541064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:32.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.679+0000 7f885a4b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88541069d0 con 0x7f88540f51a0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f88594b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88541064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f88594b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88541064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55664/0 (socket says 172.21.15.170:55664) 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f88594b5700 1 -- 172.21.15.170:0/2281319248 learned_addr learned my addr 172.21.15.170:0/2281319248 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f88594b5700 1 -- 172.21.15.170:0/2281319248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8844005040 con 0x7f88540f51a0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f88594b5700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88541064e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8844013510 tx=0x7f884400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8844009320 con 0x7f88540f51a0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8854106bd0 con 0x7f88540f51a0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8844013f00 con 0x7f88540f51a0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8844014dc0 con 0x7f88540f51a0 2023-12-08T10:07:32.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.680+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8854107060 con 0x7f88540f51a0 2023-12-08T10:07:32.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.681+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88440095f0 con 0x7f88540f51a0 2023-12-08T10:07:32.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.681+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8854045ad0 con 0x7f88540f51a0 2023-12-08T10:07:32.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.681+0000 7f8849ffb700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f884002ff60 0x7f8840032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:32.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.681+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88440410e0 con 0x7f88540f51a0 2023-12-08T10:07:32.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.682+0000 7f8858cb4700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f884002ff60 0x7f8840032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:32.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.682+0000 7f8858cb4700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f884002ff60 0x7f8840032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f88500096a0 tx=0x7f8850006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:32.689 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.687+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8844014420 con 0x7f88540f51a0 2023-12-08T10:07:32.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.892+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8854107eb0 con 0x7f88540f51a0 2023-12-08T10:07:32.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.892+0000 7f8849ffb700 1 -- 172.21.15.170:0/2281319248 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f884401c920 con 0x7f88540f51a0 2023-12-08T10:07:32.895 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:32.895 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:32.896 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f884002ff60 msgr2=0x7f8840032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:32.896 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f884002ff60 0x7f8840032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f88500096a0 tx=0x7f8850006b40).stop 2023-12-08T10:07:32.896 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 msgr2=0x7f88541064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:32.896 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88541064e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8844013510 tx=0x7f884400b040).stop 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 shutdown_connections 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f884002ff60 0x7f8840032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 --2- 172.21.15.170:0/2281319248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88540f51a0 0x7f88541064e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 >> 172.21.15.170:0/2281319248 conn(0x7f88540f09a0 msgr2=0x7f885405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 shutdown_connections 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:32.894+0000 7f885a4b7700 1 -- 172.21.15.170:0/2281319248 wait complete. 2023-12-08T10:07:32.897 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:34.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:32.314622+0000 mgr.smithi023.tgiigg (mgr.14164) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:34.183 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:32.894025+0000 mon.smithi023 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.170:0/2281319248' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:34.289 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:34.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 -- 172.21.15.170:0/594375124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 msgr2=0x7f31940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 --2- 172.21.15.170:0/594375124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f31940f6930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f3184009fb0 tx=0x7f3184009b30).stop 2023-12-08T10:07:35.427 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 -- 172.21.15.170:0/594375124 shutdown_connections 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 --2- 172.21.15.170:0/594375124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f31940f6930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 -- 172.21.15.170:0/594375124 >> 172.21.15.170:0/594375124 conn(0x7f31940f1c70 msgr2=0x7f31940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 -- 172.21.15.170:0/594375124 shutdown_connections 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.423+0000 7f319a393700 1 -- 172.21.15.170:0/594375124 wait complete. 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f319a393700 1 Processor -- start 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f319a393700 1 -- start start 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f319a393700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f3194106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f319a393700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3194106c20 con 0x7f31940f6510 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f3193fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f3194106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f3193fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f3194106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55682/0 (socket says 172.21.15.170:55682) 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.424+0000 7f3193fff700 1 -- 172.21.15.170:0/1590540194 learned_addr learned my addr 172.21.15.170:0/1590540194 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:35.428 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f3193fff700 1 -- 172.21.15.170:0/1590540194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3184005040 con 0x7f31940f6510 2023-12-08T10:07:35.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f3193fff700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f3194106730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f3184013510 tx=0x7f318400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:35.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3184009320 con 0x7f31940f6510 2023-12-08T10:07:35.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3194106e20 con 0x7f31940f6510 2023-12-08T10:07:35.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3184013f00 con 0x7f31940f6510 2023-12-08T10:07:35.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3184014d90 con 0x7f31940f6510 2023-12-08T10:07:35.429 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.425+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31941072b0 con 0x7f31940f6510 2023-12-08T10:07:35.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.426+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3184014420 con 0x7f31940f6510 2023-12-08T10:07:35.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.426+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3194045ad0 con 0x7f31940f6510 2023-12-08T10:07:35.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.426+0000 7f3190ff9700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f317c02ff60 0x7f317c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:35.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.426+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3184041070 con 0x7f31940f6510 2023-12-08T10:07:35.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.426+0000 7f31937fe700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f317c02ff60 0x7f317c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:35.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.426+0000 7f31937fe700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f317c02ff60 0x7f317c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f31880096a0 tx=0x7f3188006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:35.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.431+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3184026bf0 con 0x7f31940f6510 2023-12-08T10:07:35.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.640+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31940fbf60 con 0x7f31940f6510 2023-12-08T10:07:35.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f3190ff9700 1 -- 172.21.15.170:0/1590540194 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f318401db70 con 0x7f31940f6510 2023-12-08T10:07:35.643 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:35.643 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f317c02ff60 msgr2=0x7f317c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f317c02ff60 0x7f317c032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f31880096a0 tx=0x7f3188006b40).stop 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 msgr2=0x7f3194106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f3194106730 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f3184013510 tx=0x7f318400b040).stop 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 shutdown_connections 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f317c02ff60 0x7f317c032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 --2- 172.21.15.170:0/1590540194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31940f6510 0x7f3194106730 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.642+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 >> 172.21.15.170:0/1590540194 conn(0x7f31940f1c70 msgr2=0x7f31940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.643+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 shutdown_connections 2023-12-08T10:07:35.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:35.643+0000 7f319a393700 1 -- 172.21.15.170:0/1590540194 wait complete. 2023-12-08T10:07:35.646 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:36.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:34.315001+0000 mgr.smithi023.tgiigg (mgr.14164) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:36.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:35.642410+0000 mon.smithi023 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.170:0/1590540194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:37.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:37.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:38.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:36.315299+0000 mgr.smithi023.tgiigg (mgr.14164) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:38.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 -- 172.21.15.170:0/3504563612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 msgr2=0x7f66000f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:38.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 --2- 172.21.15.170:0/3504563612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f66000f5d70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f65e8009fb0 tx=0x7f65e8009b30).stop 2023-12-08T10:07:38.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 -- 172.21.15.170:0/3504563612 shutdown_connections 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 --2- 172.21.15.170:0/3504563612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f66000f5d70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 -- 172.21.15.170:0/3504563612 >> 172.21.15.170:0/3504563612 conn(0x7f66000f0260 msgr2=0x7f66000f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 -- 172.21.15.170:0/3504563612 shutdown_connections 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.190+0000 7f6605dc0700 1 -- 172.21.15.170:0/3504563612 wait complete. 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f6605dc0700 1 Processor -- start 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f6605dc0700 1 -- start start 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f6605dc0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f660010b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:38.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f6605dc0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001071c0 con 0x7f66000f5950 2023-12-08T10:07:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f65ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f660010b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f65ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f660010b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55698/0 (socket says 172.21.15.170:55698) 2023-12-08T10:07:38.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f65ff7fe700 1 -- 172.21.15.170:0/3478440127 learned_addr learned my addr 172.21.15.170:0/3478440127 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:38.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.191+0000 7f65ff7fe700 1 -- 172.21.15.170:0/3478440127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8005040 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.192+0000 7f65ff7fe700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f660010b7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f65e8013510 tx=0x7f65e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.192+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65e8009320 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.192+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66000f5370 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.192+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65e8013f00 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.192+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65e8014d90 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.192+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f660010bcc0 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.193+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65e80095f0 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.193+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6600045ad0 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.193+0000 7f65f7fff700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f65ec02ff60 0x7f65ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.193+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65e8041000 con 0x7f66000f5950 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.193+0000 7f65feffd700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f65ec02ff60 0x7f65ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:38.196 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.193+0000 7f65feffd700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f65ec02ff60 0x7f65ec032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f65f00096a0 tx=0x7f65f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:38.198 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.197+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65e8014420 con 0x7f66000f5950 2023-12-08T10:07:38.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.405+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f660010c4e0 con 0x7f66000f5950 2023-12-08T10:07:38.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.405+0000 7f65f7fff700 1 -- 172.21.15.170:0/3478440127 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65e801d7f0 con 0x7f66000f5950 2023-12-08T10:07:38.407 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:38.407 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f65ec02ff60 msgr2=0x7f65ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f65ec02ff60 0x7f65ec032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f65f00096a0 tx=0x7f65f0006b40).stop 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 msgr2=0x7f660010b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f660010b7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f65e8013510 tx=0x7f65e800b040).stop 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 shutdown_connections 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f65ec02ff60 0x7f65ec032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 --2- 172.21.15.170:0/3478440127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66000f5950 0x7f660010b7d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 >> 172.21.15.170:0/3478440127 conn(0x7f66000f0260 msgr2=0x7f66000f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 shutdown_connections 2023-12-08T10:07:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:38.407+0000 7f6605dc0700 1 -- 172.21.15.170:0/3478440127 wait complete. 2023-12-08T10:07:38.410 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:39.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:38 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:38.406063+0000 mon.smithi023 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.170:0/3478440127' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:39.795 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:39.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:40.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:38.315587+0000 mgr.smithi023.tgiigg (mgr.14164) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2760755055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 msgr2=0x7fb2cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 --2- 172.21.15.170:0/2760755055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc0f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb2bc009fb0 tx=0x7fb2bc009b30).stop 2023-12-08T10:07:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2760755055 shutdown_connections 2023-12-08T10:07:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 --2- 172.21.15.170:0/2760755055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc0f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:40.938 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2760755055 >> 172.21.15.170:0/2760755055 conn(0x7fb2cc0f1c70 msgr2=0x7fb2cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2760755055 shutdown_connections 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.935+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2760755055 wait complete. 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d3c6a700 1 Processor -- start 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d3c6a700 1 -- start start 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d3c6a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d3c6a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc106c20 con 0x7fb2cc0f6510 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d1a06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d1a06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55702/0 (socket says 172.21.15.170:55702) 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.936+0000 7fb2d1a06700 1 -- 172.21.15.170:0/2842476007 learned_addr learned my addr 172.21.15.170:0/2842476007 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2d1a06700 1 -- 172.21.15.170:0/2842476007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc005040 con 0x7fb2cc0f6510 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2d1a06700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fb2bc014000 tx=0x7fb2bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2bc0095f0 con 0x7fb2cc0f6510 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2bc007e70 con 0x7fb2cc0f6510 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc106e20 con 0x7fb2cc0f6510 2023-12-08T10:07:40.939 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2bc014ea0 con 0x7fb2cc0f6510 2023-12-08T10:07:40.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.937+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc1072b0 con 0x7fb2cc0f6510 2023-12-08T10:07:40.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.938+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2bc0079c0 con 0x7fb2cc0f6510 2023-12-08T10:07:40.940 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.938+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc045ad0 con 0x7fb2cc0f6510 2023-12-08T10:07:40.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.938+0000 7fb2c27fc700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb2b802ff60 0x7fb2b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:40.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.938+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2bc040920 con 0x7fb2cc0f6510 2023-12-08T10:07:40.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.938+0000 7fb2d1205700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb2b802ff60 0x7fb2b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:40.941 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.939+0000 7fb2d1205700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb2b802ff60 0x7fb2b8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb2c80096a0 tx=0x7fb2c8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:40.943+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2bc00f950 con 0x7fb2cc0f6510 2023-12-08T10:07:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.148+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2cc0fbf60 con 0x7fb2cc0f6510 2023-12-08T10:07:41.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.149+0000 7fb2c27fc700 1 -- 172.21.15.170:0/2842476007 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb2bc010230 con 0x7fb2cc0f6510 2023-12-08T10:07:41.150 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:41.151 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb2b802ff60 msgr2=0x7fb2b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb2b802ff60 0x7fb2b8032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fb2c80096a0 tx=0x7fb2c8006b40).stop 2023-12-08T10:07:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 msgr2=0x7fb2cc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fb2bc014000 tx=0x7fb2bc00b040).stop 2023-12-08T10:07:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 shutdown_connections 2023-12-08T10:07:41.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb2b802ff60 0x7fb2b8032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 --2- 172.21.15.170:0/2842476007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2cc0f6510 0x7fb2cc106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 >> 172.21.15.170:0/2842476007 conn(0x7fb2cc0f1c70 msgr2=0x7fb2cc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 shutdown_connections 2023-12-08T10:07:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:41.151+0000 7fb2d3c6a700 1 -- 172.21.15.170:0/2842476007 wait complete. 2023-12-08T10:07:41.153 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:42.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:40.315912+0000 mgr.smithi023.tgiigg (mgr.14164) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:42.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:41.149668+0000 mon.smithi023 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.170:0/2842476007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:42.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:42.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:43.896 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3117626460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 msgr2=0x7f24f80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 --2- 172.21.15.170:0/3117626460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f80f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f24ec009fb0 tx=0x7f24ec009b30).stop 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3117626460 shutdown_connections 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 --2- 172.21.15.170:0/3117626460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f80f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3117626460 >> 172.21.15.170:0/3117626460 conn(0x7f24f80f09a0 msgr2=0x7f24f80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3117626460 shutdown_connections 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.894+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3117626460 wait complete. 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.895+0000 7f24fc8e3700 1 Processor -- start 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.895+0000 7f24fc8e3700 1 -- start start 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.895+0000 7f24fc8e3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.895+0000 7f24fc8e3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f8060240 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24f6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24f6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55930/0 (socket says 172.21.15.170:55930) 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24f6d9d700 1 -- 172.21.15.170:0/3979297713 learned_addr learned my addr 172.21.15.170:0/3979297713 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24f6d9d700 1 -- 172.21.15.170:0/3979297713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec005040 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24f6d9d700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f81091b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f24ec009830 tx=0x7f24ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24ec009320 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24ec0136c0 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f8060440 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24ec01ee90 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.896+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f8109860 con 0x7f24f80f51a0 2023-12-08T10:07:43.899 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.897+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24ec014440 con 0x7f24f80f51a0 2023-12-08T10:07:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.897+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8045ad0 con 0x7f24f80f51a0 2023-12-08T10:07:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.897+0000 7f24df7fe700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f24e002ff10 0x7f24e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.897+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24ec041330 con 0x7f24f80f51a0 2023-12-08T10:07:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.898+0000 7f24f659c700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f24e002ff10 0x7f24e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:43.900 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.898+0000 7f24f659c700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f24e002ff10 0x7f24e00323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f24e80096a0 tx=0x7f24e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:43.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:43.902+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24ec01c910 con 0x7f24f80f51a0 2023-12-08T10:07:44.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.104+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24f810a080 con 0x7f24f80f51a0 2023-12-08T10:07:44.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.105+0000 7f24df7fe700 1 -- 172.21.15.170:0/3979297713 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24ec01c320 con 0x7f24f80f51a0 2023-12-08T10:07:44.107 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:44.107 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f24e002ff10 msgr2=0x7f24e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f24e002ff10 0x7f24e00323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f24e80096a0 tx=0x7f24e8006b40).stop 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 msgr2=0x7f24f81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f81091b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f24ec009830 tx=0x7f24ec00b040).stop 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 shutdown_connections 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f24e002ff10 0x7f24e00323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 --2- 172.21.15.170:0/3979297713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80f51a0 0x7f24f81091b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 >> 172.21.15.170:0/3979297713 conn(0x7f24f80f09a0 msgr2=0x7f24f80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:44.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 shutdown_connections 2023-12-08T10:07:44.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:44.107+0000 7f24fc8e3700 1 -- 172.21.15.170:0/3979297713 wait complete. 2023-12-08T10:07:44.109 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:42.316295+0000 mgr.smithi023.tgiigg (mgr.14164) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:45.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:44 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:44.105915+0000 mon.smithi023 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.170:0/3979297713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:45.493 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:45.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:44.316654+0000 mgr.smithi023.tgiigg (mgr.14164) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:46.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/410481896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 msgr2=0x7fa8c40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:46.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 --2- 172.21.15.170:0/410481896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c40f90f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fa8b8009fb0 tx=0x7fa8b8009b30).stop 2023-12-08T10:07:46.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/410481896 shutdown_connections 2023-12-08T10:07:46.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 --2- 172.21.15.170:0/410481896 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c40f90f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:46.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/410481896 >> 172.21.15.170:0/410481896 conn(0x7fa8c40f0b70 msgr2=0x7fa8c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:46.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/410481896 shutdown_connections 2023-12-08T10:07:46.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/410481896 wait complete. 2023-12-08T10:07:46.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.605+0000 7fa8c9ebf700 1 Processor -- start 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c9ebf700 1 -- start start 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c9ebf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c9ebf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4106bd0 con 0x7fa8c4060280 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55946/0 (socket says 172.21.15.170:55946) 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c37fe700 1 -- 172.21.15.170:0/1655919236 learned_addr learned my addr 172.21.15.170:0/1655919236 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.606+0000 7fa8c37fe700 1 -- 172.21.15.170:0/1655919236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8005040 con 0x7fa8c4060280 2023-12-08T10:07:46.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8c37fe700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c41066e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa8b8013510 tx=0x7fa8b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8009320 con 0x7fa8c4060280 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4106dd0 con 0x7fa8c4060280 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8b8013f00 con 0x7fa8c4060280 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8014dc0 con 0x7fa8c4060280 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c4107260 con 0x7fa8c4060280 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.607+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8b8014420 con 0x7fa8c4060280 2023-12-08T10:07:46.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.608+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4045ad0 con 0x7fa8c4060280 2023-12-08T10:07:46.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.608+0000 7fa8abfff700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa8ac02ff60 0x7fa8ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:46.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.608+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8b8041030 con 0x7fa8c4060280 2023-12-08T10:07:46.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.608+0000 7fa8c2ffd700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa8ac02ff60 0x7fa8ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:46.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.608+0000 7fa8c2ffd700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa8ac02ff60 0x7fa8ac032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:46.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.612+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8b80095f0 con 0x7fa8c4060280 2023-12-08T10:07:46.820 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.819+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8c4108110 con 0x7fa8c4060280 2023-12-08T10:07:46.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.820+0000 7fa8abfff700 1 -- 172.21.15.170:0/1655919236 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8b801cb10 con 0x7fa8c4060280 2023-12-08T10:07:46.821 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:46.821 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa8ac02ff60 msgr2=0x7fa8ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa8ac02ff60 0x7fa8ac032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa8b40096a0 tx=0x7fa8b4006b40).stop 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 msgr2=0x7fa8c41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c41066e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa8b8013510 tx=0x7fa8b800b040).stop 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 shutdown_connections 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa8ac02ff60 0x7fa8ac032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 --2- 172.21.15.170:0/1655919236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8c4060280 0x7fa8c41066e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.821+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 >> 172.21.15.170:0/1655919236 conn(0x7fa8c40f0b70 msgr2=0x7fa8c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.822+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 shutdown_connections 2023-12-08T10:07:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:46.822+0000 7fa8c9ebf700 1 -- 172.21.15.170:0/1655919236 wait complete. 2023-12-08T10:07:46.824 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:47.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:46 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:46.820459+0000 mon.smithi023 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.170:0/1655919236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:48.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:46.316957+0000 mgr.smithi023.tgiigg (mgr.14164) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:48.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:48.237 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:49.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1757098627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 msgr2=0x7f0b980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:49.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 --2- 172.21.15.170:0/1757098627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b980f66c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f0b94009fb0 tx=0x7f0b94009b30).stop 2023-12-08T10:07:49.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1757098627 shutdown_connections 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 --2- 172.21.15.170:0/1757098627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b980f66c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1757098627 >> 172.21.15.170:0/1757098627 conn(0x7f0b980f1aa0 msgr2=0x7f0b980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1757098627 shutdown_connections 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1757098627 wait complete. 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 Processor -- start 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.607+0000 7f0b9e3b0700 1 -- start start 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9e3b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b9806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9e3b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b98069e20 con 0x7f0b980f62a0 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9d3ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b9806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9d3ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b9806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:55972/0 (socket says 172.21.15.170:55972) 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9d3ae700 1 -- 172.21.15.170:0/1870730916 learned_addr learned my addr 172.21.15.170:0/1870730916 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9d3ae700 1 -- 172.21.15.170:0/1870730916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94005040 con 0x7f0b980f62a0 2023-12-08T10:07:49.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9d3ae700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b9806b9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0b94013510 tx=0x7f0b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:49.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b94009320 con 0x7f0b980f62a0 2023-12-08T10:07:49.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b98069fc0 con 0x7f0b980f62a0 2023-12-08T10:07:49.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b94013f00 con 0x7f0b980f62a0 2023-12-08T10:07:49.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.608+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b94014dc0 con 0x7f0b980f62a0 2023-12-08T10:07:49.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.609+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9806a450 con 0x7f0b980f62a0 2023-12-08T10:07:49.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.609+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b94014420 con 0x7f0b980f62a0 2023-12-08T10:07:49.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.609+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b98059100 con 0x7f0b980f62a0 2023-12-08T10:07:49.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.609+0000 7f0b8dffb700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8402ffb0 0x7f0b84032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:49.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.610+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b94041100 con 0x7f0b980f62a0 2023-12-08T10:07:49.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.610+0000 7f0b9cbad700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8402ffb0 0x7f0b84032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:49.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.610+0000 7f0b9cbad700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8402ffb0 0x7f0b84032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0b880096a0 tx=0x7f0b88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:49.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.614+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b940095f0 con 0x7f0b980f62a0 2023-12-08T10:07:49.820 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.819+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b98045ad0 con 0x7f0b980f62a0 2023-12-08T10:07:49.820 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.819+0000 7f0b8dffb700 1 -- 172.21.15.170:0/1870730916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b9401db70 con 0x7f0b980f62a0 2023-12-08T10:07:49.821 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:49.821 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:49.822 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8402ffb0 msgr2=0x7f0b84032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:49.822 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8402ffb0 0x7f0b84032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0b880096a0 tx=0x7f0b88006b40).stop 2023-12-08T10:07:49.822 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 msgr2=0x7f0b9806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:49.822 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b9806b9a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0b94013510 tx=0x7f0b9400b040).stop 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 shutdown_connections 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8402ffb0 0x7f0b84032470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 --2- 172.21.15.170:0/1870730916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b980f62a0 0x7f0b9806b9a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 >> 172.21.15.170:0/1870730916 conn(0x7f0b980f1aa0 msgr2=0x7f0b980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 shutdown_connections 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:49.821+0000 7f0b9e3b0700 1 -- 172.21.15.170:0/1870730916 wait complete. 2023-12-08T10:07:49.823 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:48.317330+0000 mgr.smithi023.tgiigg (mgr.14164) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:49.819941+0000 mon.smithi023 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.170:0/1870730916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:51.210 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:51.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:50.317627+0000 mgr.smithi023.tgiigg (mgr.14164) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:52.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 -- 172.21.15.170:0/3159436534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 msgr2=0x7fa5100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 --2- 172.21.15.170:0/3159436534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa500009fb0 tx=0x7fa500009b30).stop 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 -- 172.21.15.170:0/3159436534 shutdown_connections 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 --2- 172.21.15.170:0/3159436534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 -- 172.21.15.170:0/3159436534 >> 172.21.15.170:0/3159436534 conn(0x7fa5100f0b70 msgr2=0x7fa5100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 -- 172.21.15.170:0/3159436534 shutdown_connections 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.410+0000 7fa517370700 1 -- 172.21.15.170:0/3159436534 wait complete. 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa517370700 1 Processor -- start 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa517370700 1 -- start start 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa517370700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa517370700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100fb820 con 0x7fa5100f5410 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa51510c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa51510c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40978/0 (socket says 172.21.15.170:40978) 2023-12-08T10:07:52.413 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa51510c700 1 -- 172.21.15.170:0/1158605503 learned_addr learned my addr 172.21.15.170:0/1158605503 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:52.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa51510c700 1 -- 172.21.15.170:0/1158605503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500005040 con 0x7fa5100f5410 2023-12-08T10:07:52.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.411+0000 7fa51510c700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa500013510 tx=0x7fa50000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.412+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa500009320 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.412+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5100fb9c0 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.412+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa500013f00 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.412+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa500014d90 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.412+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5100fbe50 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.413+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5000263a0 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.413+0000 7fa505ffb700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4fc02ffb0 0x7fa4fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.413+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa500041eb0 con 0x7fa5100f5410 2023-12-08T10:07:52.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.413+0000 7fa51490b700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4fc02ffb0 0x7fa4fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:52.416 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.414+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa510059100 con 0x7fa5100f5410 2023-12-08T10:07:52.416 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.414+0000 7fa51490b700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4fc02ffb0 0x7fa4fc032470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa50c0096a0 tx=0x7fa50c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:52.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.419+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa500012920 con 0x7fa5100f5410 2023-12-08T10:07:52.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.623+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5100fc4b0 con 0x7fa5100f5410 2023-12-08T10:07:52.624 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.623+0000 7fa505ffb700 1 -- 172.21.15.170:0/1158605503 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa50002b020 con 0x7fa5100f5410 2023-12-08T10:07:52.625 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:52.625 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:52.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4fc02ffb0 msgr2=0x7fa4fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:52.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4fc02ffb0 0x7fa4fc032470 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fa50c0096a0 tx=0x7fa50c006b40).stop 2023-12-08T10:07:52.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 msgr2=0x7fa5100fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:52.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa500013510 tx=0x7fa50000b040).stop 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 shutdown_connections 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4fc02ffb0 0x7fa4fc032470 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 --2- 172.21.15.170:0/1158605503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5100f5410 0x7fa5100fd3a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 >> 172.21.15.170:0/1158605503 conn(0x7fa5100f0b70 msgr2=0x7fa51005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 shutdown_connections 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:52.625+0000 7fa517370700 1 -- 172.21.15.170:0/1158605503 wait complete. 2023-12-08T10:07:52.627 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:53.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:52.623957+0000 mon.smithi023 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.170:0/1158605503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:54.005 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:54.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:52.317963+0000 mgr.smithi023.tgiigg (mgr.14164) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- 172.21.15.170:0/395614061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 msgr2=0x7faf780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 --2- 172.21.15.170:0/395614061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf780f66c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7faf74009fb0 tx=0x7faf74009b30).stop 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- 172.21.15.170:0/395614061 shutdown_connections 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 --2- 172.21.15.170:0/395614061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf780f66c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- 172.21.15.170:0/395614061 >> 172.21.15.170:0/395614061 conn(0x7faf780f1aa0 msgr2=0x7faf780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- 172.21.15.170:0/395614061 shutdown_connections 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- 172.21.15.170:0/395614061 wait complete. 2023-12-08T10:07:55.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 Processor -- start 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- start start 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf781064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.197+0000 7faf7e62f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf781069a0 con 0x7faf780f62a0 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7d62d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf781064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7d62d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf781064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40994/0 (socket says 172.21.15.170:40994) 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7d62d700 1 -- 172.21.15.170:0/1777283591 learned_addr learned my addr 172.21.15.170:0/1777283591 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:55.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7d62d700 1 -- 172.21.15.170:0/1777283591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74005040 con 0x7faf780f62a0 2023-12-08T10:07:55.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7d62d700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf781064b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7faf74013510 tx=0x7faf7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:55.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74009320 con 0x7faf780f62a0 2023-12-08T10:07:55.201 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf78106ba0 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faf74013f00 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74014dc0 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.198+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf78107030 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.199+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf740095f0 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.199+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf78045ad0 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.199+0000 7faf6dffb700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faf6402ff60 0x7faf64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.199+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf74041020 con 0x7faf780f62a0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.200+0000 7faf7ce2c700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faf6402ff60 0x7faf64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:55.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.200+0000 7faf7ce2c700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faf6402ff60 0x7faf64032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7faf680096a0 tx=0x7faf68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:55.205 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.204+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf74014420 con 0x7faf780f62a0 2023-12-08T10:07:55.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.411+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf78107ee0 con 0x7faf780f62a0 2023-12-08T10:07:55.412 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.411+0000 7faf6dffb700 1 -- 172.21.15.170:0/1777283591 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf7401c920 con 0x7faf780f62a0 2023-12-08T10:07:55.413 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:55.413 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:55.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faf6402ff60 msgr2=0x7faf64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:55.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faf6402ff60 0x7faf64032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7faf680096a0 tx=0x7faf68006b40).stop 2023-12-08T10:07:55.414 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 msgr2=0x7faf781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf781064b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7faf74013510 tx=0x7faf7400b040).stop 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 shutdown_connections 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faf6402ff60 0x7faf64032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 --2- 172.21.15.170:0/1777283591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf780f62a0 0x7faf781064b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 >> 172.21.15.170:0/1777283591 conn(0x7faf780f1aa0 msgr2=0x7faf780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 shutdown_connections 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:55.413+0000 7faf7e62f700 1 -- 172.21.15.170:0/1777283591 wait complete. 2023-12-08T10:07:55.415 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:54.318361+0000 mgr.smithi023.tgiigg (mgr.14164) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:55.411861+0000 mon.smithi023 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.170:0/1777283591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:56.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:56.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 -- 172.21.15.170:0/4111760363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 msgr2=0x7fbb300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 --2- 172.21.15.170:0/4111760363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb300f6930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fbb20009fb0 tx=0x7fbb20009b30).stop 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 -- 172.21.15.170:0/4111760363 shutdown_connections 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 --2- 172.21.15.170:0/4111760363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb300f6930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 -- 172.21.15.170:0/4111760363 >> 172.21.15.170:0/4111760363 conn(0x7fbb300f1c70 msgr2=0x7fbb300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 -- 172.21.15.170:0/4111760363 shutdown_connections 2023-12-08T10:07:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.973+0000 7fbb384a0700 1 -- 172.21.15.170:0/4111760363 wait complete. 2023-12-08T10:07:57.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb384a0700 1 Processor -- start 2023-12-08T10:07:57.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb384a0700 1 -- start start 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb384a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb384a0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30106c20 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb3623c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb3623c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41006/0 (socket says 172.21.15.170:41006) 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.974+0000 7fbb3623c700 1 -- 172.21.15.170:0/1779274396 learned_addr learned my addr 172.21.15.170:0/1779274396 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb3623c700 1 -- 172.21.15.170:0/1779274396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb20005040 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb3623c700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb30106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fbb20013510 tx=0x7fbb2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb20009320 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb20013f00 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb30106e20 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb20014d90 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.975+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb301072b0 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.976+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb200263a0 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.976+0000 7fbb26ffd700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbb1c02ff60 0x7fbb1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.976+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb20041080 con 0x7fbb300f6510 2023-12-08T10:07:57.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.976+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb30045ad0 con 0x7fbb300f6510 2023-12-08T10:07:57.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.976+0000 7fbb35a3b700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbb1c02ff60 0x7fbb1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:07:57.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.977+0000 7fbb35a3b700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbb1c02ff60 0x7fbb1c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbb2c0096a0 tx=0x7fbb2c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:07:57.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:57.981+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb20012920 con 0x7fbb300f6510 2023-12-08T10:07:58.190 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.188+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb300fc050 con 0x7fbb300f6510 2023-12-08T10:07:58.191 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.189+0000 7fbb26ffd700 1 -- 172.21.15.170:0/1779274396 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb2002b020 con 0x7fbb300f6510 2023-12-08T10:07:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:07:58.191 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbb1c02ff60 msgr2=0x7fbb1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbb1c02ff60 0x7fbb1c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fbb2c0096a0 tx=0x7fbb2c006b40).stop 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 msgr2=0x7fbb30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb30106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fbb20013510 tx=0x7fbb2000b040).stop 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 shutdown_connections 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbb1c02ff60 0x7fbb1c032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:58.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 --2- 172.21.15.170:0/1779274396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb300f6510 0x7fbb30106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:07:58.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 >> 172.21.15.170:0/1779274396 conn(0x7fbb300f1c70 msgr2=0x7fbb300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:07:58.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 shutdown_connections 2023-12-08T10:07:58.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:07:58.191+0000 7fbb384a0700 1 -- 172.21.15.170:0/1779274396 wait complete. 2023-12-08T10:07:58.194 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:07:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:56.318657+0000 mgr.smithi023.tgiigg (mgr.14164) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:07:59.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:07:58.190538+0000 mon.smithi023 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.170:0/1779274396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:07:59.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:07:59.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:07:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:07:58.318891+0000 mgr.smithi023.tgiigg (mgr.14164) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:00.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 -- 172.21.15.170:0/1325557728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 msgr2=0x7f88a80f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:00.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 --2- 172.21.15.170:0/1325557728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f4c00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f889c009fb0 tx=0x7f889c009b30).stop 2023-12-08T10:08:00.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 -- 172.21.15.170:0/1325557728 shutdown_connections 2023-12-08T10:08:00.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 --2- 172.21.15.170:0/1325557728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f4c00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:00.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 -- 172.21.15.170:0/1325557728 >> 172.21.15.170:0/1325557728 conn(0x7f88a80f0260 msgr2=0x7f88a80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 -- 172.21.15.170:0/1325557728 shutdown_connections 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.698+0000 7f88ad151700 1 -- 172.21.15.170:0/1325557728 wait complete. 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88ad151700 1 Processor -- start 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88ad151700 1 -- start start 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88ad151700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88ad151700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80f5850 con 0x7f88a80f47e0 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88a6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88a6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41018/0 (socket says 172.21.15.170:41018) 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.699+0000 7f88a6d9d700 1 -- 172.21.15.170:0/3526910264 learned_addr learned my addr 172.21.15.170:0/3526910264 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f88a6d9d700 1 -- 172.21.15.170:0/3526910264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c005040 con 0x7f88a80f47e0 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f88a6d9d700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f73d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f889c013510 tx=0x7f889c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:00.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c009320 con 0x7f88a80f47e0 2023-12-08T10:08:00.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a80f59f0 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f889c013f00 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f889c014dc0 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.700+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a80f5e80 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.701+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f889c014420 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.701+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a8059100 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.701+0000 7f888f7fe700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f889002ff60 0x7f8890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.701+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f889c041050 con 0x7f88a80f47e0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.701+0000 7f88a659c700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f889002ff60 0x7f8890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:00.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.701+0000 7f88a659c700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f889002ff60 0x7f8890032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f88980096a0 tx=0x7f8898006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:00.708 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.706+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f889c0095f0 con 0x7f88a80f47e0 2023-12-08T10:08:00.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.910+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88a8045ad0 con 0x7f88a80f47e0 2023-12-08T10:08:00.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.911+0000 7f888f7fe700 1 -- 172.21.15.170:0/3526910264 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f889c01cb10 con 0x7f88a80f47e0 2023-12-08T10:08:00.914 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:00.914 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:00.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f889002ff60 msgr2=0x7f8890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:00.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f889002ff60 0x7f8890032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f88980096a0 tx=0x7f8898006b40).stop 2023-12-08T10:08:00.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 msgr2=0x7f88a80f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:00.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f73d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f889c013510 tx=0x7f889c00b040).stop 2023-12-08T10:08:00.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 shutdown_connections 2023-12-08T10:08:00.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f889002ff60 0x7f8890032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:00.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 --2- 172.21.15.170:0/3526910264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88a80f47e0 0x7f88a80f73d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:00.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 >> 172.21.15.170:0/3526910264 conn(0x7f88a80f0260 msgr2=0x7f88a80f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:00.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 shutdown_connections 2023-12-08T10:08:00.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:00.913+0000 7f88ad151700 1 -- 172.21.15.170:0/3526910264 wait complete. 2023-12-08T10:08:00.916 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:01.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:00 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:00.912901+0000 mon.smithi023 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.170:0/3526910264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:02.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:02.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:00.319168+0000 mgr.smithi023.tgiigg (mgr.14164) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:03.450 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 -- 172.21.15.170:0/1465941240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 msgr2=0x7f0a340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 --2- 172.21.15.170:0/1465941240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a340f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0a30009fb0 tx=0x7f0a30009b30).stop 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 -- 172.21.15.170:0/1465941240 shutdown_connections 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 --2- 172.21.15.170:0/1465941240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a340f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 -- 172.21.15.170:0/1465941240 >> 172.21.15.170:0/1465941240 conn(0x7f0a340f0b70 msgr2=0x7f0a340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 -- 172.21.15.170:0/1465941240 shutdown_connections 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 -- 172.21.15.170:0/1465941240 wait complete. 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 Processor -- start 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.447+0000 7f0a3c07f700 1 -- start start 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a3c07f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a34109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a3c07f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a34060240 con 0x7f0a340f5410 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a39e1b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a34109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:03.451 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a39e1b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a34109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52110/0 (socket says 172.21.15.170:52110) 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a39e1b700 1 -- 172.21.15.170:0/2825438578 learned_addr learned my addr 172.21.15.170:0/2825438578 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a39e1b700 1 -- 172.21.15.170:0/2825438578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a30005040 con 0x7f0a340f5410 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a39e1b700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a34109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f0a30013510 tx=0x7f0a3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a30009320 con 0x7f0a340f5410 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a34060440 con 0x7f0a340f5410 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a30013f00 con 0x7f0a340f5410 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.448+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a30014d90 con 0x7f0a340f5410 2023-12-08T10:08:03.452 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.449+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a34109ae0 con 0x7f0a340f5410 2023-12-08T10:08:03.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.449+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a300263a0 con 0x7f0a340f5410 2023-12-08T10:08:03.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.449+0000 7f0a2a7fc700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a2002ff60 0x7f0a20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:03.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.449+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a30040fe0 con 0x7f0a340f5410 2023-12-08T10:08:03.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.449+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a34045ad0 con 0x7f0a340f5410 2023-12-08T10:08:03.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.450+0000 7f0a3961a700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a2002ff60 0x7f0a20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:03.453 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.450+0000 7f0a3961a700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a2002ff60 0x7f0a20032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0a240096a0 tx=0x7f0a24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:03.456 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.454+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a30012920 con 0x7f0a340f5410 2023-12-08T10:08:03.663 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.661+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a3410a230 con 0x7f0a340f5410 2023-12-08T10:08:03.664 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.661+0000 7f0a2a7fc700 1 -- 172.21.15.170:0/2825438578 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a3002b020 con 0x7f0a340f5410 2023-12-08T10:08:03.664 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:03.664 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:03.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a2002ff60 msgr2=0x7f0a20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:03.665 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a2002ff60 0x7f0a20032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0a240096a0 tx=0x7f0a24006b40).stop 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 msgr2=0x7f0a34109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a34109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f0a30013510 tx=0x7f0a3000b040).stop 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 shutdown_connections 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a2002ff60 0x7f0a20032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 --2- 172.21.15.170:0/2825438578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a340f5410 0x7f0a34109430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 >> 172.21.15.170:0/2825438578 conn(0x7f0a340f0b70 msgr2=0x7f0a340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 shutdown_connections 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:03.663+0000 7f0a3c07f700 1 -- 172.21.15.170:0/2825438578 wait complete. 2023-12-08T10:08:03.666 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:02.319511+0000 mgr.smithi023.tgiigg (mgr.14164) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:04.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:03.662879+0000 mon.smithi023 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.170:0/2825438578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:05.050 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:05.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 -- 172.21.15.170:0/3158467369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 msgr2=0x7f013c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 --2- 172.21.15.170:0/3158467369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c0f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f0124009fb0 tx=0x7f0124009b30).stop 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 -- 172.21.15.170:0/3158467369 shutdown_connections 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 --2- 172.21.15.170:0/3158467369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c0f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 -- 172.21.15.170:0/3158467369 >> 172.21.15.170:0/3158467369 conn(0x7f013c0f0b70 msgr2=0x7f013c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 -- 172.21.15.170:0/3158467369 shutdown_connections 2023-12-08T10:08:06.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.225+0000 7f01419b4700 1 -- 172.21.15.170:0/3158467369 wait complete. 2023-12-08T10:08:06.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f01419b4700 1 Processor -- start 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f01419b4700 1 -- start start 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f01419b4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f01419b4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c060280 con 0x7f013c0f5410 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f013affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f013affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52134/0 (socket says 172.21.15.170:52134) 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.226+0000 7f013affd700 1 -- 172.21.15.170:0/2042045047 learned_addr learned my addr 172.21.15.170:0/2042045047 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f013affd700 1 -- 172.21.15.170:0/2042045047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124005040 con 0x7f013c0f5410 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f013affd700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c109390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0124013510 tx=0x7f012400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0124009320 con 0x7f013c0f5410 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c060480 con 0x7f013c0f5410 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0124013f00 con 0x7f013c0f5410 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0124014d90 con 0x7f013c0f5410 2023-12-08T10:08:06.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.227+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c109ac0 con 0x7f013c0f5410 2023-12-08T10:08:06.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.228+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01240095f0 con 0x7f013c0f5410 2023-12-08T10:08:06.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.228+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013c045ad0 con 0x7f013c0f5410 2023-12-08T10:08:06.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.228+0000 7f01337fe700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f012802ff60 0x7f0128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:06.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.228+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01240410c0 con 0x7f013c0f5410 2023-12-08T10:08:06.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.228+0000 7f013a7fc700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f012802ff60 0x7f0128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:06.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.229+0000 7f013a7fc700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f012802ff60 0x7f0128032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f012c0096a0 tx=0x7f012c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:06.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.233+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0124014420 con 0x7f013c0f5410 2023-12-08T10:08:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:04.319830+0000 mgr.smithi023.tgiigg (mgr.14164) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:06.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.439+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f013c0fae60 con 0x7f013c0f5410 2023-12-08T10:08:06.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.439+0000 7f01337fe700 1 -- 172.21.15.170:0/2042045047 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0124010230 con 0x7f013c0f5410 2023-12-08T10:08:06.442 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:06.442 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f012802ff60 msgr2=0x7f0128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f012802ff60 0x7f0128032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f012c0096a0 tx=0x7f012c006b40).stop 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 msgr2=0x7f013c109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c109390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f0124013510 tx=0x7f012400b040).stop 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 shutdown_connections 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f012802ff60 0x7f0128032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 --2- 172.21.15.170:0/2042045047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f013c0f5410 0x7f013c109390 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:06.443 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 >> 172.21.15.170:0/2042045047 conn(0x7f013c0f0b70 msgr2=0x7f013c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:06.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 shutdown_connections 2023-12-08T10:08:06.444 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:06.441+0000 7f01419b4700 1 -- 172.21.15.170:0/2042045047 wait complete. 2023-12-08T10:08:06.444 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:07.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:06.440872+0000 mon.smithi023 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.170:0/2042045047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:07.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:07.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:06.320102+0000 mgr.smithi023.tgiigg (mgr.14164) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 -- 172.21.15.170:0/1260601482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 msgr2=0x7fe4a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 --2- 172.21.15.170:0/1260601482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a00f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe494009fb0 tx=0x7fe494009b30).stop 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 -- 172.21.15.170:0/1260601482 shutdown_connections 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 --2- 172.21.15.170:0/1260601482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a00f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 -- 172.21.15.170:0/1260601482 >> 172.21.15.170:0/1260601482 conn(0x7fe4a00f0b70 msgr2=0x7fe4a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 -- 172.21.15.170:0/1260601482 shutdown_connections 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.013+0000 7fe4a4e59700 1 -- 172.21.15.170:0/1260601482 wait complete. 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe4a4e59700 1 Processor -- start 2023-12-08T10:08:09.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe4a4e59700 1 -- start start 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe4a4e59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe4a4e59700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0106c00 con 0x7fe4a00f5410 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe49e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe49e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52138/0 (socket says 172.21.15.170:52138) 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe49e59c700 1 -- 172.21.15.170:0/2980002659 learned_addr learned my addr 172.21.15.170:0/2980002659 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe49e59c700 1 -- 172.21.15.170:0/2980002659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494005040 con 0x7fe4a00f5410 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.014+0000 7fe49e59c700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a0106710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe494013510 tx=0x7fe49400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe494009320 con 0x7fe4a00f5410 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a0106e00 con 0x7fe4a00f5410 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe494013f00 con 0x7fe4a00f5410 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe494014d90 con 0x7fe4a00f5410 2023-12-08T10:08:09.018 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a0107290 con 0x7fe4a00f5410 2023-12-08T10:08:09.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4940095f0 con 0x7fe4a00f5410 2023-12-08T10:08:09.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.015+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a0045ad0 con 0x7fe4a00f5410 2023-12-08T10:08:09.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.016+0000 7fe48effd700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe48802ff60 0x7fe488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:09.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.016+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe494040fc0 con 0x7fe4a00f5410 2023-12-08T10:08:09.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.016+0000 7fe49dd9b700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe48802ff60 0x7fe488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:09.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.016+0000 7fe49dd9b700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe48802ff60 0x7fe488032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe4900096a0 tx=0x7fe490006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:09.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.020+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe494014420 con 0x7fe4a00f5410 2023-12-08T10:08:09.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.227+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4a00fae60 con 0x7fe4a00f5410 2023-12-08T10:08:09.230 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.228+0000 7fe48effd700 1 -- 172.21.15.170:0/2980002659 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe494010230 con 0x7fe4a00f5410 2023-12-08T10:08:09.230 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:09.230 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe48802ff60 msgr2=0x7fe488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:09.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe48802ff60 0x7fe488032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe4900096a0 tx=0x7fe490006b40).stop 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 msgr2=0x7fe4a0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a0106710 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fe494013510 tx=0x7fe49400b040).stop 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 shutdown_connections 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe48802ff60 0x7fe488032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 --2- 172.21.15.170:0/2980002659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4a00f5410 0x7fe4a0106710 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 >> 172.21.15.170:0/2980002659 conn(0x7fe4a00f0b70 msgr2=0x7fe4a005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 shutdown_connections 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:09.229+0000 7fe4a4e59700 1 -- 172.21.15.170:0/2980002659 wait complete. 2023-12-08T10:08:09.232 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:08.320388+0000 mgr.smithi023.tgiigg (mgr.14164) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:09.229371+0000 mon.smithi023 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.170:0/2980002659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:10.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:10.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:11.782 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.779+0000 7fd679f39700 1 -- 172.21.15.170:0/1510150701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 msgr2=0x7fd6740f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.779+0000 7fd679f39700 1 --2- 172.21.15.170:0/1510150701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd6740f66c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fd668009fb0 tx=0x7fd668009b30).stop 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 -- 172.21.15.170:0/1510150701 shutdown_connections 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 --2- 172.21.15.170:0/1510150701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd6740f66c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 -- 172.21.15.170:0/1510150701 >> 172.21.15.170:0/1510150701 conn(0x7fd6740f1aa0 msgr2=0x7fd6740f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 -- 172.21.15.170:0/1510150701 shutdown_connections 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 -- 172.21.15.170:0/1510150701 wait complete. 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 Processor -- start 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 -- start start 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd67406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:11.783 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.780+0000 7fd679f39700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd674069e20 con 0x7fd6740f62a0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd678f37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd67406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd678f37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd67406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52156/0 (socket says 172.21.15.170:52156) 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd678f37700 1 -- 172.21.15.170:0/3084016441 learned_addr learned my addr 172.21.15.170:0/3084016441 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd678f37700 1 -- 172.21.15.170:0/3084016441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd668005040 con 0x7fd6740f62a0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd678f37700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd67406b9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd668013510 tx=0x7fd66800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd668009320 con 0x7fd6740f62a0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd674069fc0 con 0x7fd6740f62a0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd668013f00 con 0x7fd6740f62a0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd668014dc0 con 0x7fd6740f62a0 2023-12-08T10:08:11.785 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.781+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67406a450 con 0x7fd6740f62a0 2023-12-08T10:08:11.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.782+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6680095f0 con 0x7fd6740f62a0 2023-12-08T10:08:11.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.782+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd674059100 con 0x7fd6740f62a0 2023-12-08T10:08:11.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.782+0000 7fd6717fa700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd66402ff60 0x7fd664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:11.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.782+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd668041020 con 0x7fd6740f62a0 2023-12-08T10:08:11.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.782+0000 7fd673fff700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd66402ff60 0x7fd664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:11.786 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.783+0000 7fd673fff700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd66402ff60 0x7fd664032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd6600096a0 tx=0x7fd660006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:11.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.787+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd668014420 con 0x7fd6740f62a0 2023-12-08T10:08:11.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.994+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd674045ad0 con 0x7fd6740f62a0 2023-12-08T10:08:11.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.994+0000 7fd6717fa700 1 -- 172.21.15.170:0/3084016441 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd66801c920 con 0x7fd6740f62a0 2023-12-08T10:08:11.997 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:11.997 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.995+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd66402ff60 msgr2=0x7fd664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.995+0000 7fd679f39700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd66402ff60 0x7fd664032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fd6600096a0 tx=0x7fd660006b40).stop 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.995+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 msgr2=0x7fd67406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.995+0000 7fd679f39700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd67406b9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fd668013510 tx=0x7fd66800b040).stop 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.996+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 shutdown_connections 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.996+0000 7fd679f39700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd66402ff60 0x7fd664032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.996+0000 7fd679f39700 1 --2- 172.21.15.170:0/3084016441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6740f62a0 0x7fd67406b9a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.996+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 >> 172.21.15.170:0/3084016441 conn(0x7fd6740f1aa0 msgr2=0x7fd6740f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.996+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 shutdown_connections 2023-12-08T10:08:11.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:11.996+0000 7fd679f39700 1 -- 172.21.15.170:0/3084016441 wait complete. 2023-12-08T10:08:11.999 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:12.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:10.320679+0000 mgr.smithi023.tgiigg (mgr.14164) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:13.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:13.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:13.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:11.995896+0000 mon.smithi023 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.170:0/3084016441' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:14.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:12.320950+0000 mgr.smithi023.tgiigg (mgr.14164) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 -- 172.21.15.170:0/1771605123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 msgr2=0x7f88fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 --2- 172.21.15.170:0/1771605123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc0f55c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f88ec009fb0 tx=0x7f88ec009b30).stop 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 -- 172.21.15.170:0/1771605123 shutdown_connections 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 --2- 172.21.15.170:0/1771605123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc0f55c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 -- 172.21.15.170:0/1771605123 >> 172.21.15.170:0/1771605123 conn(0x7f88fc0f09a0 msgr2=0x7f88fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 -- 172.21.15.170:0/1771605123 shutdown_connections 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 -- 172.21.15.170:0/1771605123 wait complete. 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.574+0000 7f89009ca700 1 Processor -- start 2023-12-08T10:08:14.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f89009ca700 1 -- start start 2023-12-08T10:08:14.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f89009ca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:14.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f89009ca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc1069d0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f88fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f88fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:38988/0 (socket says 172.21.15.170:38988) 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f88fb7fe700 1 -- 172.21.15.170:0/929230442 learned_addr learned my addr 172.21.15.170:0/929230442 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f88fb7fe700 1 -- 172.21.15.170:0/929230442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec005040 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.575+0000 7f88fb7fe700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc1064e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f88ec013510 tx=0x7f88ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.576+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ec009320 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.576+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc106bd0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.576+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88ec013f00 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.576+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc107060 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.576+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88ec014dc0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.576+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88ec01d3d0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.577+0000 7f88e3fff700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f88e402ff60 0x7f88e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.577+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88ec0410b0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.577+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc045ad0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.577+0000 7f88faffd700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f88e402ff60 0x7f88e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:14.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.577+0000 7f88faffd700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f88e402ff60 0x7f88e4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f88f00096a0 tx=0x7f88f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:14.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.582+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88ec012920 con 0x7f88fc0f51a0 2023-12-08T10:08:14.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.787+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88fc107da0 con 0x7f88fc0f51a0 2023-12-08T10:08:14.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.788+0000 7f88e3fff700 1 -- 172.21.15.170:0/929230442 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f88ec012330 con 0x7f88fc0f51a0 2023-12-08T10:08:14.793 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:14.794 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f88e402ff60 msgr2=0x7f88e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f88e402ff60 0x7f88e4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f88f00096a0 tx=0x7f88f0006b40).stop 2023-12-08T10:08:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 msgr2=0x7f88fc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:14.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc1064e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f88ec013510 tx=0x7f88ec00b040).stop 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 shutdown_connections 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f88e402ff60 0x7f88e4032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 --2- 172.21.15.170:0/929230442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88fc0f51a0 0x7f88fc1064e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 >> 172.21.15.170:0/929230442 conn(0x7f88fc0f09a0 msgr2=0x7f88fc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 shutdown_connections 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:14.789+0000 7f89009ca700 1 -- 172.21.15.170:0/929230442 wait complete. 2023-12-08T10:08:14.795 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:15.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:14.789420+0000 mon.smithi023 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.170:0/929230442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:16.183 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:16.183 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:16.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:14.321278+0000 mgr.smithi023.tgiigg (mgr.14164) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 -- 172.21.15.170:0/102447912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 msgr2=0x7f91f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 --2- 172.21.15.170:0/102447912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f40f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f91e8009fb0 tx=0x7f91e8009b30).stop 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 -- 172.21.15.170:0/102447912 shutdown_connections 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 --2- 172.21.15.170:0/102447912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f40f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 -- 172.21.15.170:0/102447912 >> 172.21.15.170:0/102447912 conn(0x7f91f40f0b70 msgr2=0x7f91f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 -- 172.21.15.170:0/102447912 shutdown_connections 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.313+0000 7f91fa54b700 1 -- 172.21.15.170:0/102447912 wait complete. 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.314+0000 7f91fa54b700 1 Processor -- start 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.314+0000 7f91fa54b700 1 -- start start 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.314+0000 7f91fa54b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:17.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.314+0000 7f91fa54b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4060240 con 0x7f91f40f5410 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39000/0 (socket says 172.21.15.170:39000) 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f3fff700 1 -- 172.21.15.170:0/3767804112 learned_addr learned my addr 172.21.15.170:0/3767804112 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f3fff700 1 -- 172.21.15.170:0/3767804112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8005040 con 0x7f91f40f5410 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f3fff700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f4109430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f91e8013510 tx=0x7f91e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8009320 con 0x7f91f40f5410 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f4060440 con 0x7f91f40f5410 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91e8013f00 con 0x7f91f40f5410 2023-12-08T10:08:17.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.315+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8014d90 con 0x7f91f40f5410 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.316+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f4109ae0 con 0x7f91f40f5410 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.316+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91e80263a0 con 0x7f91f40f5410 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.316+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4045ad0 con 0x7f91f40f5410 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.316+0000 7f91f0ff9700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f91dc02ff60 0x7f91dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.316+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91e80410c0 con 0x7f91f40f5410 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.317+0000 7f91f37fe700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f91dc02ff60 0x7f91dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:17.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.317+0000 7f91f37fe700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f91dc02ff60 0x7f91dc032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f91e40096a0 tx=0x7f91e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:17.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.321+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91e80095f0 con 0x7f91f40f5410 2023-12-08T10:08:17.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.529+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91f40fae60 con 0x7f91f40f5410 2023-12-08T10:08:17.532 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.530+0000 7f91f0ff9700 1 -- 172.21.15.170:0/3767804112 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f91e801db70 con 0x7f91f40f5410 2023-12-08T10:08:17.533 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:17.533 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f91dc02ff60 msgr2=0x7f91dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f91dc02ff60 0x7f91dc032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f91e40096a0 tx=0x7f91e4006b40).stop 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 msgr2=0x7f91f4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f4109430 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f91e8013510 tx=0x7f91e800b040).stop 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 shutdown_connections 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f91dc02ff60 0x7f91dc032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 --2- 172.21.15.170:0/3767804112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91f40f5410 0x7f91f4109430 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 >> 172.21.15.170:0/3767804112 conn(0x7f91f40f0b70 msgr2=0x7f91f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 shutdown_connections 2023-12-08T10:08:17.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:17.532+0000 7f91fa54b700 1 -- 172.21.15.170:0/3767804112 wait complete. 2023-12-08T10:08:17.535 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:18.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:16.321579+0000 mgr.smithi023.tgiigg (mgr.14164) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:18.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:17.531853+0000 mon.smithi023 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.170:0/3767804112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:18.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:18.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:20.356 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 -- 172.21.15.170:0/2945445535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 msgr2=0x7f4e340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 --2- 172.21.15.170:0/2945445535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f90f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4e28009fb0 tx=0x7f4e28009b30).stop 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 -- 172.21.15.170:0/2945445535 shutdown_connections 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 --2- 172.21.15.170:0/2945445535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f90f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 -- 172.21.15.170:0/2945445535 >> 172.21.15.170:0/2945445535 conn(0x7f4e340f0b70 msgr2=0x7f4e340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 -- 172.21.15.170:0/2945445535 shutdown_connections 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.353+0000 7f4e3abbc700 1 -- 172.21.15.170:0/2945445535 wait complete. 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e3abbc700 1 Processor -- start 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e3abbc700 1 -- start start 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e3abbc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:20.357 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e3abbc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340f4ff0 con 0x7f4e34060280 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e38958700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e38958700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39016/0 (socket says 172.21.15.170:39016) 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.354+0000 7f4e38958700 1 -- 172.21.15.170:0/3037257323 learned_addr learned my addr 172.21.15.170:0/3037257323 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e38958700 1 -- 172.21.15.170:0/3037257323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e28005040 con 0x7f4e34060280 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e38958700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f6b70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4e28013510 tx=0x7f4e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e28009320 con 0x7f4e34060280 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340f5190 con 0x7f4e34060280 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e28013f00 con 0x7f4e34060280 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e28014dc0 con 0x7f4e34060280 2023-12-08T10:08:20.358 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.355+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e340f5620 con 0x7f4e34060280 2023-12-08T10:08:20.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.356+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e2801d3d0 con 0x7f4e34060280 2023-12-08T10:08:20.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.356+0000 7f4e317fa700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e2402ffb0 0x7f4e24032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:20.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.356+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e280410d0 con 0x7f4e34060280 2023-12-08T10:08:20.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.356+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e34059100 con 0x7f4e34060280 2023-12-08T10:08:20.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.356+0000 7f4e33fff700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e2402ffb0 0x7f4e24032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:20.359 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.357+0000 7f4e33fff700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e2402ffb0 0x7f4e24032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4e200096a0 tx=0x7f4e20006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:20.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.361+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e28026490 con 0x7f4e34060280 2023-12-08T10:08:20.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:18.321870+0000 mgr.smithi023.tgiigg (mgr.14164) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:20.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.568+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e34045ad0 con 0x7f4e34060280 2023-12-08T10:08:20.570 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.569+0000 7f4e317fa700 1 -- 172.21.15.170:0/3037257323 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e280126f0 con 0x7f4e34060280 2023-12-08T10:08:20.570 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:20.570 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:20.571 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e2402ffb0 msgr2=0x7f4e24032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e2402ffb0 0x7f4e24032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4e200096a0 tx=0x7f4e20006b40).stop 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 msgr2=0x7f4e340f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f6b70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f4e28013510 tx=0x7f4e2800b040).stop 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 shutdown_connections 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e2402ffb0 0x7f4e24032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 --2- 172.21.15.170:0/3037257323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e34060280 0x7f4e340f6b70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 >> 172.21.15.170:0/3037257323 conn(0x7f4e340f0b70 msgr2=0x7f4e340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 shutdown_connections 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:20.570+0000 7f4e3abbc700 1 -- 172.21.15.170:0/3037257323 wait complete. 2023-12-08T10:08:20.572 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:21.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:20.569242+0000 mon.smithi023 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.170:0/3037257323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:21.937 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:21.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:22.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:20.322227+0000 mgr.smithi023.tgiigg (mgr.14164) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.132+0000 7f6f169a0700 1 -- 172.21.15.170:0/2811844976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 msgr2=0x7f6f100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 --2- 172.21.15.170:0/2811844976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f100f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f6f0c009fb0 tx=0x7f6f0c009b30).stop 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 -- 172.21.15.170:0/2811844976 shutdown_connections 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 --2- 172.21.15.170:0/2811844976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f100f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 -- 172.21.15.170:0/2811844976 >> 172.21.15.170:0/2811844976 conn(0x7f6f100f09a0 msgr2=0x7f6f100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 -- 172.21.15.170:0/2811844976 shutdown_connections 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 -- 172.21.15.170:0/2811844976 wait complete. 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 Processor -- start 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.133+0000 7f6f169a0700 1 -- start start 2023-12-08T10:08:23.135 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f169a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f10109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:23.136 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f169a0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f10060280 con 0x7f6f100f51a0 2023-12-08T10:08:23.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f1599e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f10109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:23.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f1599e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f10109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60952/0 (socket says 172.21.15.170:60952) 2023-12-08T10:08:23.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f1599e700 1 -- 172.21.15.170:0/3691278813 learned_addr learned my addr 172.21.15.170:0/3691278813 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:23.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f1599e700 1 -- 172.21.15.170:0/3691278813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c005040 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f1599e700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f10109170 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6f0c013510 tx=0x7f6f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.134+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f0c009320 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f10060480 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f0c013f00 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f0c014dc0 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f101098a0 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f0c0095f0 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f10045ad0 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.135+0000 7f6f067fc700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6efc02ff60 0x7f6efc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.136+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f0c0410c0 con 0x7f6f100f51a0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.136+0000 7f6f1519d700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6efc02ff60 0x7f6efc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:23.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.136+0000 7f6f1519d700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6efc02ff60 0x7f6efc032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f6f000096a0 tx=0x7f6f00006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:23.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.140+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f0c014420 con 0x7f6f100f51a0 2023-12-08T10:08:23.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.342+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f100fabe0 con 0x7f6f100f51a0 2023-12-08T10:08:23.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.343+0000 7f6f067fc700 1 -- 172.21.15.170:0/3691278813 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f0c010230 con 0x7f6f100f51a0 2023-12-08T10:08:23.344 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:23.344 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6efc02ff60 msgr2=0x7f6efc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6efc02ff60 0x7f6efc032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f6f000096a0 tx=0x7f6f00006b40).stop 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 msgr2=0x7f6f10109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f10109170 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f6f0c013510 tx=0x7f6f0c00b040).stop 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 shutdown_connections 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6efc02ff60 0x7f6efc032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:23.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 --2- 172.21.15.170:0/3691278813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f100f51a0 0x7f6f10109170 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:23.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 >> 172.21.15.170:0/3691278813 conn(0x7f6f100f09a0 msgr2=0x7f6f100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:23.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 shutdown_connections 2023-12-08T10:08:23.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:23.345+0000 7f6f169a0700 1 -- 172.21.15.170:0/3691278813 wait complete. 2023-12-08T10:08:23.347 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:22.450270+0000 mon.smithi023 (mon.0) 152 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:08:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:22.453123+0000 mon.smithi023 (mon.0) 153 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:08:24.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:22.322549+0000 mgr.smithi023.tgiigg (mgr.14164) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:24.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:23.343726+0000 mon.smithi023 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.170:0/3691278813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:24.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:24.746 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:25.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 -- 172.21.15.170:0/1163971822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 msgr2=0x7f23f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 --2- 172.21.15.170:0/1163971822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f00f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f23e4009fb0 tx=0x7f23e4009b30).stop 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 -- 172.21.15.170:0/1163971822 shutdown_connections 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 --2- 172.21.15.170:0/1163971822 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f00f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 -- 172.21.15.170:0/1163971822 >> 172.21.15.170:0/1163971822 conn(0x7f23f00f0b70 msgr2=0x7f23f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 -- 172.21.15.170:0/1163971822 shutdown_connections 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.906+0000 7f23f517e700 1 -- 172.21.15.170:0/1163971822 wait complete. 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23f517e700 1 Processor -- start 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23f517e700 1 -- start start 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23f517e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f01093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23f517e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0060280 con 0x7f23f00f5410 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23eed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f01093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23eed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f01093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60964/0 (socket says 172.21.15.170:60964) 2023-12-08T10:08:25.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.907+0000 7f23eed9d700 1 -- 172.21.15.170:0/642124690 learned_addr learned my addr 172.21.15.170:0/642124690 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23eed9d700 1 -- 172.21.15.170:0/642124690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e4005040 con 0x7f23f00f5410 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23eed9d700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f01093c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f23e4014000 tx=0x7f23e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e40095f0 con 0x7f23f00f5410 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f0060480 con 0x7f23f00f5410 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23e4007e70 con 0x7f23f00f5410 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e4014ea0 con 0x7f23f00f5410 2023-12-08T10:08:25.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.908+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f0109af0 con 0x7f23f00f5410 2023-12-08T10:08:25.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.909+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23e40079c0 con 0x7f23f00f5410 2023-12-08T10:08:25.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.909+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f0045ad0 con 0x7f23f00f5410 2023-12-08T10:08:25.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.909+0000 7f23d77fe700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f23d802ffb0 0x7f23d8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:25.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.909+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23e4040910 con 0x7f23f00f5410 2023-12-08T10:08:25.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.909+0000 7f23ee59c700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f23d802ffb0 0x7f23d8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:25.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.910+0000 7f23ee59c700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f23d802ffb0 0x7f23d8032470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f23e00096a0 tx=0x7f23e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:25.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:25.914+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23e400f950 con 0x7f23f00f5410 2023-12-08T10:08:26.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.123+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23f010a310 con 0x7f23f00f5410 2023-12-08T10:08:26.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.123+0000 7f23d77fe700 1 -- 172.21.15.170:0/642124690 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23e4010230 con 0x7f23f00f5410 2023-12-08T10:08:26.125 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:26.125 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:26.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f23d802ffb0 msgr2=0x7f23d8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:26.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f23d802ffb0 0x7f23d8032470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f23e00096a0 tx=0x7f23e0006b40).stop 2023-12-08T10:08:26.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 msgr2=0x7f23f01093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:26.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f01093c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f23e4014000 tx=0x7f23e400b040).stop 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 shutdown_connections 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f23d802ffb0 0x7f23d8032470 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 --2- 172.21.15.170:0/642124690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f00f5410 0x7f23f01093c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 >> 172.21.15.170:0/642124690 conn(0x7f23f00f0b70 msgr2=0x7f23f00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 shutdown_connections 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:26.125+0000 7f23f517e700 1 -- 172.21.15.170:0/642124690 wait complete. 2023-12-08T10:08:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:26.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:26 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:24.322872+0000 mgr.smithi023.tgiigg (mgr.14164) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:27.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:26.124015+0000 mon.smithi023 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.170:0/642124690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:27.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:27.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:28.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:28 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:26.323166+0000 mgr.smithi023.tgiigg (mgr.14164) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:28.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 -- 172.21.15.170:0/1446656832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 msgr2=0x7f33b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:28.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 --2- 172.21.15.170:0/1446656832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b80f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f33a8009fb0 tx=0x7f33a8009b30).stop 2023-12-08T10:08:28.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 -- 172.21.15.170:0/1446656832 shutdown_connections 2023-12-08T10:08:28.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 --2- 172.21.15.170:0/1446656832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b80f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:28.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 -- 172.21.15.170:0/1446656832 >> 172.21.15.170:0/1446656832 conn(0x7f33b80f0b70 msgr2=0x7f33b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 -- 172.21.15.170:0/1446656832 shutdown_connections 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.668+0000 7f33bfd97700 1 -- 172.21.15.170:0/1446656832 wait complete. 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bfd97700 1 Processor -- start 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bfd97700 1 -- start start 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bfd97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bfd97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b8106c10 con 0x7f33b80f5410 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bdb33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bdb33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60984/0 (socket says 172.21.15.170:60984) 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.669+0000 7f33bdb33700 1 -- 172.21.15.170:0/295913400 learned_addr learned my addr 172.21.15.170:0/295913400 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33bdb33700 1 -- 172.21.15.170:0/295913400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a8005040 con 0x7f33b80f5410 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33bdb33700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b8106720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f33a8013510 tx=0x7f33a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33a8009320 con 0x7f33b80f5410 2023-12-08T10:08:28.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b8106e10 con 0x7f33b80f5410 2023-12-08T10:08:28.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33a8013f00 con 0x7f33b80f5410 2023-12-08T10:08:28.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33a8014d90 con 0x7f33b80f5410 2023-12-08T10:08:28.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.670+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b81072a0 con 0x7f33b80f5410 2023-12-08T10:08:28.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.671+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b8045ad0 con 0x7f33b80f5410 2023-12-08T10:08:28.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.671+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33a80095f0 con 0x7f33b80f5410 2023-12-08T10:08:28.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.671+0000 7f33ae7fc700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f33a402ff60 0x7f33a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:28.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.671+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33a8040fe0 con 0x7f33b80f5410 2023-12-08T10:08:28.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.671+0000 7f33bd332700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f33a402ff60 0x7f33a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:28.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.672+0000 7f33bd332700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f33a402ff60 0x7f33a4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f33b40096a0 tx=0x7f33b4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:28.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.676+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33a8010230 con 0x7f33b80f5410 2023-12-08T10:08:28.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.879+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33b81080f0 con 0x7f33b80f5410 2023-12-08T10:08:28.881 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.880+0000 7f33ae7fc700 1 -- 172.21.15.170:0/295913400 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f33a8010230 con 0x7f33b80f5410 2023-12-08T10:08:28.882 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:28.882 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:28.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f33a402ff60 msgr2=0x7f33a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f33a402ff60 0x7f33a4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f33b40096a0 tx=0x7f33b4006b40).stop 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 msgr2=0x7f33b8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b8106720 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f33a8013510 tx=0x7f33a800b040).stop 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 shutdown_connections 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f33a402ff60 0x7f33a4032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 --2- 172.21.15.170:0/295913400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33b80f5410 0x7f33b8106720 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 >> 172.21.15.170:0/295913400 conn(0x7f33b80f0b70 msgr2=0x7f33b805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 shutdown_connections 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:28.882+0000 7f33bfd97700 1 -- 172.21.15.170:0/295913400 wait complete. 2023-12-08T10:08:28.884 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:29.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:28.880993+0000 mon.smithi023 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.170:0/295913400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:30.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:30.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:30.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:28.323383+0000 mgr.smithi023.tgiigg (mgr.14164) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:31.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 -- 172.21.15.170:0/3084432212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 msgr2=0x7f47f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:31.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 --2- 172.21.15.170:0/3084432212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f80f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f47f0009fb0 tx=0x7f47f0009b30).stop 2023-12-08T10:08:31.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 -- 172.21.15.170:0/3084432212 shutdown_connections 2023-12-08T10:08:31.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 --2- 172.21.15.170:0/3084432212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f80f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:31.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 -- 172.21.15.170:0/3084432212 >> 172.21.15.170:0/3084432212 conn(0x7f47f80f1c70 msgr2=0x7f47f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:31.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 -- 172.21.15.170:0/3084432212 shutdown_connections 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.436+0000 7f47fcff5700 1 -- 172.21.15.170:0/3084432212 wait complete. 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47fcff5700 1 Processor -- start 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47fcff5700 1 -- start start 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47fcff5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47fcff5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f8069e20 con 0x7f47f80f6510 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47f659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47f659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:32772/0 (socket says 172.21.15.170:32772) 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47f659c700 1 -- 172.21.15.170:0/1677068360 learned_addr learned my addr 172.21.15.170:0/1677068360 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47f659c700 1 -- 172.21.15.170:0/1677068360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0005040 con 0x7f47f80f6510 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.437+0000 7f47f659c700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f806b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f47f0013510 tx=0x7f47f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.438+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f0009320 con 0x7f47f80f6510 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.438+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f8069fc0 con 0x7f47f80f6510 2023-12-08T10:08:31.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.438+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f0013f00 con 0x7f47f80f6510 2023-12-08T10:08:31.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.438+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f0014d90 con 0x7f47f80f6510 2023-12-08T10:08:31.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.438+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f806a450 con 0x7f47f80f6510 2023-12-08T10:08:31.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.438+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47f00263a0 con 0x7f47f80f6510 2023-12-08T10:08:31.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.439+0000 7f47eeffd700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f47e402ff60 0x7f47e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:31.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.439+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47f0040fc0 con 0x7f47f80f6510 2023-12-08T10:08:31.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.439+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f8059100 con 0x7f47f80f6510 2023-12-08T10:08:31.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.439+0000 7f47f5d9b700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f47e402ff60 0x7f47e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:31.442 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.439+0000 7f47f5d9b700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f47e402ff60 0x7f47e4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f47e00096a0 tx=0x7f47e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:31.445 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.444+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47f0012920 con 0x7f47f80f6510 2023-12-08T10:08:31.652 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.651+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47f80fbf60 con 0x7f47f80f6510 2023-12-08T10:08:31.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.651+0000 7f47eeffd700 1 -- 172.21.15.170:0/1677068360 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47f002b020 con 0x7f47f80f6510 2023-12-08T10:08:31.653 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:31.653 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f47e402ff60 msgr2=0x7f47e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f47e402ff60 0x7f47e4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f47e00096a0 tx=0x7f47e0006b40).stop 2023-12-08T10:08:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 msgr2=0x7f47f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:31.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f806b9a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f47f0013510 tx=0x7f47f000b040).stop 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 shutdown_connections 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f47e402ff60 0x7f47e4032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 --2- 172.21.15.170:0/1677068360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47f80f6510 0x7f47f806b9a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 >> 172.21.15.170:0/1677068360 conn(0x7f47f80f1c70 msgr2=0x7f47f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 shutdown_connections 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:31.653+0000 7f47fcff5700 1 -- 172.21.15.170:0/1677068360 wait complete. 2023-12-08T10:08:31.655 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:32.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:30.323685+0000 mgr.smithi023.tgiigg (mgr.14164) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:32.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:31.652124+0000 mon.smithi023 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.170:0/1677068360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:33.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:33.060 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:34.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3249232323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 msgr2=0x7fe2180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:34.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 --2- 172.21.15.170:0/3249232323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe2180f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe208009fb0 tx=0x7fe208009b30).stop 2023-12-08T10:08:34.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3249232323 shutdown_connections 2023-12-08T10:08:34.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 --2- 172.21.15.170:0/3249232323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe2180f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:34.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3249232323 >> 172.21.15.170:0/3249232323 conn(0x7fe2180f0b70 msgr2=0x7fe2180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:34.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3249232323 shutdown_connections 2023-12-08T10:08:34.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3249232323 wait complete. 2023-12-08T10:08:34.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 Processor -- start 2023-12-08T10:08:34.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.214+0000 7fe21f9b0700 1 -- start start 2023-12-08T10:08:34.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21f9b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe218105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:34.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21f9b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe218105b20 con 0x7fe2180f5410 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21d74c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe218105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21d74c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe218105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41564/0 (socket says 172.21.15.170:41564) 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21d74c700 1 -- 172.21.15.170:0/3944354940 learned_addr learned my addr 172.21.15.170:0/3944354940 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21d74c700 1 -- 172.21.15.170:0/3944354940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208005040 con 0x7fe2180f5410 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.215+0000 7fe21d74c700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe218105630 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe208013510 tx=0x7fe20800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.216+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe208009320 con 0x7fe2180f5410 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.216+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3944354940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe218105d20 con 0x7fe2180f5410 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.216+0000 7fe21f9b0700 1 -- 172.21.15.170:0/3944354940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2181061b0 con 0x7fe2180f5410 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.216+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe208013f00 con 0x7fe2180f5410 2023-12-08T10:08:34.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.216+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe208014d60 con 0x7fe2180f5410 2023-12-08T10:08:34.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.217+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe208014420 con 0x7fe2180f5410 2023-12-08T10:08:34.219 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.217+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe218045ad0 con 0x7fe2180f5410 2023-12-08T10:08:34.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.217+0000 7fe20dffb700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe20402ff60 0x7fe204032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:34.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.217+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2080412d0 con 0x7fe2180f5410 2023-12-08T10:08:34.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.217+0000 7fe21cf4b700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe20402ff60 0x7fe204032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:34.220 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.218+0000 7fe21cf4b700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe20402ff60 0x7fe204032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:34.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.222+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe20801c320 con 0x7fe2180f5410 2023-12-08T10:08:34.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:34 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:32.324058+0000 mgr.smithi023.tgiigg (mgr.14164) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:34.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.432+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2180fae60 con 0x7fe2180f5410 2023-12-08T10:08:34.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.432+0000 7fe20dffb700 1 -- 172.21.15.170:0/3944354940 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe208012b20 con 0x7fe2180f5410 2023-12-08T10:08:34.434 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:34.434 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:34.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe20402ff60 msgr2=0x7fe204032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:34.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe20402ff60 0x7fe204032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).stop 2023-12-08T10:08:34.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 msgr2=0x7fe218105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:34.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe218105630 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe208013510 tx=0x7fe20800b040).stop 2023-12-08T10:08:34.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 shutdown_connections 2023-12-08T10:08:34.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe20402ff60 0x7fe204032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:34.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 --2- 172.21.15.170:0/3944354940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2180f5410 0x7fe218105630 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:34.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 >> 172.21.15.170:0/3944354940 conn(0x7fe2180f0b70 msgr2=0x7fe2180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:34.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 shutdown_connections 2023-12-08T10:08:34.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:34.434+0000 7fe2037fe700 1 -- 172.21.15.170:0/3944354940 wait complete. 2023-12-08T10:08:34.437 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:35.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:34.433129+0000 mon.smithi023 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.170:0/3944354940' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:35.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:35.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:36.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:36 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:34.324326+0000 mgr.smithi023.tgiigg (mgr.14164) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- 172.21.15.170:0/1671479610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 msgr2=0x7f0bf00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 --2- 172.21.15.170:0/1671479610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f8f20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0be0009fb0 tx=0x7f0be0009b30).stop 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- 172.21.15.170:0/1671479610 shutdown_connections 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 --2- 172.21.15.170:0/1671479610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f8f20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- 172.21.15.170:0/1671479610 >> 172.21.15.170:0/1671479610 conn(0x7f0bf00f09a0 msgr2=0x7f0bf00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- 172.21.15.170:0/1671479610 shutdown_connections 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- 172.21.15.170:0/1671479610 wait complete. 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 Processor -- start 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- start start 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:37.017 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.015+0000 7f0bf547b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf00f4e20 con 0x7f0bf00601e0 2023-12-08T10:08:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0beffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0beffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41576/0 (socket says 172.21.15.170:41576) 2023-12-08T10:08:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0beffff700 1 -- 172.21.15.170:0/4176469022 learned_addr learned my addr 172.21.15.170:0/4176469022 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0beffff700 1 -- 172.21.15.170:0/4176469022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be0005040 con 0x7f0bf00601e0 2023-12-08T10:08:37.019 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0beffff700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f69a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0be0014000 tx=0x7f0be000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0be00095f0 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf00f4fc0 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0be0007e70 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0be0014ea0 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.016+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf00f5450 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.017+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0be001d4e0 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.017+0000 7f0becff9700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0bd802ff60 0x7f0bd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.017+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0be00418c0 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.017+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf0059100 con 0x7f0bf00601e0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.018+0000 7f0bef7fe700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0bd802ff60 0x7f0bd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:37.020 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.018+0000 7f0bef7fe700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0bd802ff60 0x7f0bd8032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0be40096a0 tx=0x7f0be4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:37.023 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.022+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0be000f8f0 con 0x7f0bf00601e0 2023-12-08T10:08:37.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.227+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bf00f5b50 con 0x7f0bf00601e0 2023-12-08T10:08:37.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.228+0000 7f0becff9700 1 -- 172.21.15.170:0/4176469022 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0be002c090 con 0x7f0bf00601e0 2023-12-08T10:08:37.229 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:37.229 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0bd802ff60 msgr2=0x7f0bd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0bd802ff60 0x7f0bd8032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0be40096a0 tx=0x7f0be4006b40).stop 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 msgr2=0x7f0bf00f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f69a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f0be0014000 tx=0x7f0be000b040).stop 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 shutdown_connections 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0bd802ff60 0x7f0bd8032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:37.231 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 --2- 172.21.15.170:0/4176469022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf00601e0 0x7f0bf00f69a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:37.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 >> 172.21.15.170:0/4176469022 conn(0x7f0bf00f09a0 msgr2=0x7f0bf00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:37.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 shutdown_connections 2023-12-08T10:08:37.232 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:37.230+0000 7f0bf547b700 1 -- 172.21.15.170:0/4176469022 wait complete. 2023-12-08T10:08:37.232 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:38.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:38 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:36.324614+0000 mgr.smithi023.tgiigg (mgr.14164) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:38.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:38 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:37.228510+0000 mon.smithi023 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.170:0/4176469022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:38.629 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:38.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:39.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 -- 172.21.15.170:0/4266818075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 msgr2=0x7f02d00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:39.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 --2- 172.21.15.170:0/4266818075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d00f90f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f02b8009fb0 tx=0x7f02b8009b30).stop 2023-12-08T10:08:39.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 -- 172.21.15.170:0/4266818075 shutdown_connections 2023-12-08T10:08:39.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 --2- 172.21.15.170:0/4266818075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d00f90f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:39.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 -- 172.21.15.170:0/4266818075 >> 172.21.15.170:0/4266818075 conn(0x7f02d00f0b70 msgr2=0x7f02d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:39.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 -- 172.21.15.170:0/4266818075 shutdown_connections 2023-12-08T10:08:39.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.771+0000 7f02d643f700 1 -- 172.21.15.170:0/4266818075 wait complete. 2023-12-08T10:08:39.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02d643f700 1 Processor -- start 2023-12-08T10:08:39.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02d643f700 1 -- start start 2023-12-08T10:08:39.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02d643f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d01088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02d643f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0108de0 con 0x7f02d0060280 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d01088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02cffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d01088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41602/0 (socket says 172.21.15.170:41602) 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.772+0000 7f02cffff700 1 -- 172.21.15.170:0/4219489715 learned_addr learned my addr 172.21.15.170:0/4219489715 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02cffff700 1 -- 172.21.15.170:0/4219489715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b8005040 con 0x7f02d0060280 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02cffff700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d01088f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f02b8013510 tx=0x7f02b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02b8009320 con 0x7f02d0060280 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d0108fe0 con 0x7f02d0060280 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02b8013f00 con 0x7f02d0060280 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02b8014dc0 con 0x7f02d0060280 2023-12-08T10:08:39.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.773+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d0109470 con 0x7f02d0060280 2023-12-08T10:08:39.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.774+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02b8014420 con 0x7f02d0060280 2023-12-08T10:08:39.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.774+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d0045ad0 con 0x7f02d0060280 2023-12-08T10:08:39.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.774+0000 7f02ccff9700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f02bc02ff60 0x7f02bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:39.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.774+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02b8041110 con 0x7f02d0060280 2023-12-08T10:08:39.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.774+0000 7f02cf7fe700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f02bc02ff60 0x7f02bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:39.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.775+0000 7f02cf7fe700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f02bc02ff60 0x7f02bc032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f02c00096a0 tx=0x7f02c0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:39.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.779+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02b80095f0 con 0x7f02d0060280 2023-12-08T10:08:39.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.984+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02d00fbf50 con 0x7f02d0060280 2023-12-08T10:08:39.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.985+0000 7f02ccff9700 1 -- 172.21.15.170:0/4219489715 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f02b801cb10 con 0x7f02d0060280 2023-12-08T10:08:39.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:39.986 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f02bc02ff60 msgr2=0x7f02bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f02bc02ff60 0x7f02bc032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f02c00096a0 tx=0x7f02c0006b40).stop 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 msgr2=0x7f02d01088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d01088f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f02b8013510 tx=0x7f02b800b040).stop 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 shutdown_connections 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f02bc02ff60 0x7f02bc032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 --2- 172.21.15.170:0/4219489715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02d0060280 0x7f02d01088f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 >> 172.21.15.170:0/4219489715 conn(0x7f02d00f0b70 msgr2=0x7f02d00f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 shutdown_connections 2023-12-08T10:08:39.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:39.986+0000 7f02d643f700 1 -- 172.21.15.170:0/4219489715 wait complete. 2023-12-08T10:08:39.989 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:40.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:38.324961+0000 mgr.smithi023.tgiigg (mgr.14164) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:40.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:39.985324+0000 mon.smithi023 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.170:0/4219489715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:41.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:41.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:42.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:42 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:40.325255+0000 mgr.smithi023.tgiigg (mgr.14164) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.533+0000 7f142ef01700 1 -- 172.21.15.170:0/2354924513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 msgr2=0x7f14280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.533+0000 7f142ef01700 1 --2- 172.21.15.170:0/2354924513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14280f55c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f1424009fb0 tx=0x7f1424009b30).stop 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.533+0000 7f142ef01700 1 -- 172.21.15.170:0/2354924513 shutdown_connections 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.533+0000 7f142ef01700 1 --2- 172.21.15.170:0/2354924513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14280f55c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.533+0000 7f142ef01700 1 -- 172.21.15.170:0/2354924513 >> 172.21.15.170:0/2354924513 conn(0x7f14280f09a0 msgr2=0x7f14280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.534+0000 7f142ef01700 1 -- 172.21.15.170:0/2354924513 shutdown_connections 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.534+0000 7f142ef01700 1 -- 172.21.15.170:0/2354924513 wait complete. 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.534+0000 7f142ef01700 1 Processor -- start 2023-12-08T10:08:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.534+0000 7f142ef01700 1 -- start start 2023-12-08T10:08:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.534+0000 7f142ef01700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14281091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.534+0000 7f142ef01700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428060240 con 0x7f14280f51a0 2023-12-08T10:08:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f142deff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14281091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f142deff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14281091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34600/0 (socket says 172.21.15.170:34600) 2023-12-08T10:08:42.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f142deff700 1 -- 172.21.15.170:0/3987264630 learned_addr learned my addr 172.21.15.170:0/3987264630 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f142deff700 1 -- 172.21.15.170:0/3987264630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424005040 con 0x7f14280f51a0 2023-12-08T10:08:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f142deff700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14281091b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1424013510 tx=0x7f142400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1424009320 con 0x7f14280f51a0 2023-12-08T10:08:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1428060440 con 0x7f14280f51a0 2023-12-08T10:08:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1424013f00 con 0x7f14280f51a0 2023-12-08T10:08:42.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.535+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1424014dc0 con 0x7f14280f51a0 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.536+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1428109860 con 0x7f14280f51a0 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.536+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14240095f0 con 0x7f14280f51a0 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.536+0000 7f141e7fc700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f141402ff60 0x7f1414032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.536+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1428045ad0 con 0x7f14280f51a0 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.536+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14240410c0 con 0x7f14280f51a0 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.537+0000 7f142d6fe700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f141402ff60 0x7f1414032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:42.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.537+0000 7f142d6fe700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f141402ff60 0x7f1414032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f14180096a0 tx=0x7f1418006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:42.543 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.541+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f142401c790 con 0x7f14280f51a0 2023-12-08T10:08:42.749 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.747+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f142810a080 con 0x7f14280f51a0 2023-12-08T10:08:42.750 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.748+0000 7f141e7fc700 1 -- 172.21.15.170:0/3987264630 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f142402c090 con 0x7f14280f51a0 2023-12-08T10:08:42.750 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:42.750 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:42.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f141402ff60 msgr2=0x7f1414032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:42.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f141402ff60 0x7f1414032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f14180096a0 tx=0x7f1418006b40).stop 2023-12-08T10:08:42.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 msgr2=0x7f14281091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:42.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14281091b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f1424013510 tx=0x7f142400b040).stop 2023-12-08T10:08:42.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 shutdown_connections 2023-12-08T10:08:42.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f141402ff60 0x7f1414032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:42.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 --2- 172.21.15.170:0/3987264630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14280f51a0 0x7f14281091b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:42.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 >> 172.21.15.170:0/3987264630 conn(0x7f14280f09a0 msgr2=0x7f14280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:42.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 shutdown_connections 2023-12-08T10:08:42.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:42.749+0000 7f142ef01700 1 -- 172.21.15.170:0/3987264630 wait complete. 2023-12-08T10:08:42.752 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:43.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:42.325480+0000 mgr.smithi023.tgiigg (mgr.14164) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:43.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:42.749417+0000 mon.smithi023 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.170:0/3987264630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:44.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:44.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 -- 172.21.15.170:0/1388274059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 msgr2=0x7f5b1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 --2- 172.21.15.170:0/1388274059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f5b14009960 tx=0x7f5b14004d10).stop 2023-12-08T10:08:45.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 -- 172.21.15.170:0/1388274059 shutdown_connections 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 --2- 172.21.15.170:0/1388274059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 -- 172.21.15.170:0/1388274059 >> 172.21.15.170:0/1388274059 conn(0x7f5b1c0f0b70 msgr2=0x7f5b1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 -- 172.21.15.170:0/1388274059 shutdown_connections 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.291+0000 7f5b217e4700 1 -- 172.21.15.170:0/1388274059 wait complete. 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b217e4700 1 Processor -- start 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b217e4700 1 -- start start 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b217e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b217e4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0fb820 con 0x7f5b1c0f5410 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b1affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b1affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34618/0 (socket says 172.21.15.170:34618) 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.292+0000 7f5b1affd700 1 -- 172.21.15.170:0/2436300712 learned_addr learned my addr 172.21.15.170:0/2436300712 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b1affd700 1 -- 172.21.15.170:0/2436300712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14009640 con 0x7f5b1c0f5410 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b1affd700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5b14004b70 tx=0x7f5b14004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b14018070 con 0x7f5b1c0f5410 2023-12-08T10:08:45.296 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c0fb9c0 con 0x7f5b1c0f5410 2023-12-08T10:08:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b1400b870 con 0x7f5b1c0f5410 2023-12-08T10:08:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b1401c560 con 0x7f5b1c0f5410 2023-12-08T10:08:45.297 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.293+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c0fbe50 con 0x7f5b1c0f5410 2023-12-08T10:08:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.294+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b1401c6c0 con 0x7f5b1c0f5410 2023-12-08T10:08:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.294+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c059100 con 0x7f5b1c0f5410 2023-12-08T10:08:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.294+0000 7f5b037fe700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b0402ff60 0x7f5b04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.294+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b140411c0 con 0x7f5b1c0f5410 2023-12-08T10:08:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.294+0000 7f5b1a7fc700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b0402ff60 0x7f5b04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.294+0000 7f5b1a7fc700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b0402ff60 0x7f5b04032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f5b100096a0 tx=0x7f5b10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:45.301 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.299+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b14004810 con 0x7f5b1c0f5410 2023-12-08T10:08:45.507 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.505+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b1c045ad0 con 0x7f5b1c0f5410 2023-12-08T10:08:45.508 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.505+0000 7f5b037fe700 1 -- 172.21.15.170:0/2436300712 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b14004e10 con 0x7f5b1c0f5410 2023-12-08T10:08:45.508 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:45.508 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:45.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b0402ff60 msgr2=0x7f5b04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:45.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b0402ff60 0x7f5b04032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f5b100096a0 tx=0x7f5b10006b40).stop 2023-12-08T10:08:45.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 msgr2=0x7f5b1c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:45.509 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0fd3a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f5b14004b70 tx=0x7f5b14004d10).stop 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 shutdown_connections 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5b0402ff60 0x7f5b04032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 --2- 172.21.15.170:0/2436300712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c0f5410 0x7f5b1c0fd3a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 >> 172.21.15.170:0/2436300712 conn(0x7f5b1c0f0b70 msgr2=0x7f5b1c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 shutdown_connections 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:45.507+0000 7f5b217e4700 1 -- 172.21.15.170:0/2436300712 wait complete. 2023-12-08T10:08:45.510 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:45.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:44.325794+0000 mgr.smithi023.tgiigg (mgr.14164) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:46.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:46 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:45.507098+0000 mon.smithi023 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.170:0/2436300712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:46.902 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:46.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:47.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:46.326118+0000 mgr.smithi023.tgiigg (mgr.14164) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:48.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 -- 172.21.15.170:0/3356265460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd00f6510 msgr2=0x7fbcd00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:48.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 --2- 172.21.15.170:0/3356265460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd00f6510 0x7fbcd00f6930 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fbcc0009fb0 tx=0x7fbcc0009b30).stop 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 -- 172.21.15.170:0/3356265460 shutdown_connections 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 --2- 172.21.15.170:0/3356265460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd00f6510 0x7fbcd00f6930 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 -- 172.21.15.170:0/3356265460 >> 172.21.15.170:0/3356265460 conn(0x7fbcd00f1c70 msgr2=0x7fbcd00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 -- 172.21.15.170:0/3356265460 shutdown_connections 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.051+0000 7fbcd64d0700 1 -- 172.21.15.170:0/3356265460 wait complete. 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbcd64d0700 1 Processor -- start 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbcd64d0700 1 -- start start 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbcd64d0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 0x7fbcd0106c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbcd64d0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd0107150 con 0x7fbcd0106840 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbccffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 0x7fbcd0106c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbccffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 0x7fbcd0106c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34636/0 (socket says 172.21.15.170:34636) 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.052+0000 7fbccffff700 1 -- 172.21.15.170:0/1592851911 learned_addr learned my addr 172.21.15.170:0/1592851911 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbccffff700 1 -- 172.21.15.170:0/1592851911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc0005040 con 0x7fbcd0106840 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbccffff700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 0x7fbcd0106c60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fbcc0009f80 tx=0x7fbcc0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:48.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc00136c0 con 0x7fbcd0106840 2023-12-08T10:08:48.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd0107350 con 0x7fbcd0106840 2023-12-08T10:08:48.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcc00095f0 con 0x7fbcd0106840 2023-12-08T10:08:48.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc0014420 con 0x7fbcd0106840 2023-12-08T10:08:48.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.053+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd0107ea0 con 0x7fbcd0106840 2023-12-08T10:08:48.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.054+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcc0025060 con 0x7fbcd0106840 2023-12-08T10:08:48.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.054+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd0059100 con 0x7fbcd0106840 2023-12-08T10:08:48.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.054+0000 7fbcccff9700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:48.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.054+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcc0040e20 con 0x7fbcd0106840 2023-12-08T10:08:48.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.054+0000 7fbccf7fe700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:48.058 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.055+0000 7fbccf7fe700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbcb802ff60 0x7fbcb8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:48.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.059+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcc001cb60 con 0x7fbcd0106840 2023-12-08T10:08:48.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.266+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcd0045ad0 con 0x7fbcd0106840 2023-12-08T10:08:48.270 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.266+0000 7fbcccff9700 1 -- 172.21.15.170:0/1592851911 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbcc000f280 con 0x7fbcd0106840 2023-12-08T10:08:48.270 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:48.270 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:48.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbcb802ff60 msgr2=0x7fbcb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:48.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbcb802ff60 0x7fbcb8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbcc40096a0 tx=0x7fbcc4006b40).stop 2023-12-08T10:08:48.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 msgr2=0x7fbcd0106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:48.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 0x7fbcd0106c60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fbcc0009f80 tx=0x7fbcc0009b30).stop 2023-12-08T10:08:48.272 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 shutdown_connections 2023-12-08T10:08:48.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbcb802ff60 0x7fbcb8032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:48.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 --2- 172.21.15.170:0/1592851911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcd0106840 0x7fbcd0106c60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:48.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 >> 172.21.15.170:0/1592851911 conn(0x7fbcd00f1c70 msgr2=0x7fbcd00f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:48.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 shutdown_connections 2023-12-08T10:08:48.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:48.268+0000 7fbcd64d0700 1 -- 172.21.15.170:0/1592851911 wait complete. 2023-12-08T10:08:48.273 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:49.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:48 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:48.268102+0000 mon.smithi023 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.170:0/1592851911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:49.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:49.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:48.326460+0000 mgr.smithi023.tgiigg (mgr.14164) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:50.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.822+0000 7fae1366a700 1 -- 172.21.15.170:0/3727716310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 msgr2=0x7fae0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:50.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.822+0000 7fae1366a700 1 --2- 172.21.15.170:0/3727716310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c0f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fae00009fb0 tx=0x7fae00009b30).stop 2023-12-08T10:08:50.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 -- 172.21.15.170:0/3727716310 shutdown_connections 2023-12-08T10:08:50.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 --2- 172.21.15.170:0/3727716310 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c0f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:50.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 -- 172.21.15.170:0/3727716310 >> 172.21.15.170:0/3727716310 conn(0x7fae0c0f0b70 msgr2=0x7fae0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:50.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 -- 172.21.15.170:0/3727716310 shutdown_connections 2023-12-08T10:08:50.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 -- 172.21.15.170:0/3727716310 wait complete. 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 Processor -- start 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 -- start start 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae1366a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c106c10 con 0x7fae0c0f5410 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae11406700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae11406700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34668/0 (socket says 172.21.15.170:34668) 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.823+0000 7fae11406700 1 -- 172.21.15.170:0/875564587 learned_addr learned my addr 172.21.15.170:0/875564587 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fae11406700 1 -- 172.21.15.170:0/875564587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00005040 con 0x7fae0c0f5410 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fae11406700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c106720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fae00013510 tx=0x7fae0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae00009320 con 0x7fae0c0f5410 2023-12-08T10:08:50.829 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c106e10 con 0x7fae0c0f5410 2023-12-08T10:08:50.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c1072a0 con 0x7fae0c0f5410 2023-12-08T10:08:50.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae00013f00 con 0x7fae0c0f5410 2023-12-08T10:08:50.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.824+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae00014d60 con 0x7fae0c0f5410 2023-12-08T10:08:50.830 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.825+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c045ad0 con 0x7fae0c0f5410 2023-12-08T10:08:50.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.826+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae000095f0 con 0x7fae0c0f5410 2023-12-08T10:08:50.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.827+0000 7fadfdffb700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fadf802ff60 0x7fadf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:50.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.827+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae000411e0 con 0x7fae0c0f5410 2023-12-08T10:08:50.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.827+0000 7fae10c05700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fadf802ff60 0x7fadf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:50.831 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.827+0000 7fae10c05700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fadf802ff60 0x7fadf8032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fae080096a0 tx=0x7fae08006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:50.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:50.830+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae00014420 con 0x7fae0c0f5410 2023-12-08T10:08:51.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.036+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae0c108150 con 0x7fae0c0f5410 2023-12-08T10:08:51.039 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.036+0000 7fadfdffb700 1 -- 172.21.15.170:0/875564587 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae0002b070 con 0x7fae0c0f5410 2023-12-08T10:08:51.039 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:51.040 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:51.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fadf802ff60 msgr2=0x7fadf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:51.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fadf802ff60 0x7fadf8032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fae080096a0 tx=0x7fae08006b40).stop 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 msgr2=0x7fae0c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c106720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fae00013510 tx=0x7fae0000b040).stop 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 shutdown_connections 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fadf802ff60 0x7fadf8032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 --2- 172.21.15.170:0/875564587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae0c0f5410 0x7fae0c106720 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 >> 172.21.15.170:0/875564587 conn(0x7fae0c0f0b70 msgr2=0x7fae0c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 shutdown_connections 2023-12-08T10:08:51.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:51.038+0000 7fae1366a700 1 -- 172.21.15.170:0/875564587 wait complete. 2023-12-08T10:08:51.042 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:50.326789+0000 mgr.smithi023.tgiigg (mgr.14164) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:51.038105+0000 mon.smithi023 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.170:0/875564587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:52.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:52.426 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:53.601 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 -- 172.21.15.170:0/2392758671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 msgr2=0x7f7e440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:53.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 --2- 172.21.15.170:0/2392758671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e440f55c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f7e38009fb0 tx=0x7f7e38009b30).stop 2023-12-08T10:08:53.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 -- 172.21.15.170:0/2392758671 shutdown_connections 2023-12-08T10:08:53.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 --2- 172.21.15.170:0/2392758671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e440f55c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:53.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 -- 172.21.15.170:0/2392758671 >> 172.21.15.170:0/2392758671 conn(0x7f7e440f09a0 msgr2=0x7f7e440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:53.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 -- 172.21.15.170:0/2392758671 shutdown_connections 2023-12-08T10:08:53.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 -- 172.21.15.170:0/2392758671 wait complete. 2023-12-08T10:08:53.604 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.599+0000 7f7e43fff700 1 Processor -- start 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e43fff700 1 -- start start 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e43fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e441053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e43fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e441058a0 con 0x7f7e440f51a0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e42ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e441053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e42ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e441053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33934/0 (socket says 172.21.15.170:33934) 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e42ffd700 1 -- 172.21.15.170:0/4203570394 learned_addr learned my addr 172.21.15.170:0/4203570394 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e42ffd700 1 -- 172.21.15.170:0/4203570394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38005040 con 0x7f7e440f51a0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.600+0000 7f7e42ffd700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e441053b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7e38013510 tx=0x7f7e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.601+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e38009320 con 0x7f7e440f51a0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.601+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e38013f00 con 0x7f7e440f51a0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.601+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e44105aa0 con 0x7f7e440f51a0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.601+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e38014dc0 con 0x7f7e440f51a0 2023-12-08T10:08:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.601+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e44105f30 con 0x7f7e440f51a0 2023-12-08T10:08:53.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.601+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e38014420 con 0x7f7e440f51a0 2023-12-08T10:08:53.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.602+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e44045ad0 con 0x7f7e440f51a0 2023-12-08T10:08:53.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.602+0000 7f7e2b7fe700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7e2c02ff60 0x7f7e2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:53.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.602+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e38041050 con 0x7f7e440f51a0 2023-12-08T10:08:53.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.602+0000 7f7e427fc700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7e2c02ff60 0x7f7e2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:53.606 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.602+0000 7f7e427fc700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7e2c02ff60 0x7f7e2c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f7e340096a0 tx=0x7f7e34006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:53.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.606+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e38026bf0 con 0x7f7e440f51a0 2023-12-08T10:08:53.814 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.812+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e44106de0 con 0x7f7e440f51a0 2023-12-08T10:08:53.815 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.812+0000 7f7e2b7fe700 1 -- 172.21.15.170:0/4203570394 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e3801cb10 con 0x7f7e440f51a0 2023-12-08T10:08:53.815 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:53.815 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7e2c02ff60 msgr2=0x7f7e2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7e2c02ff60 0x7f7e2c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f7e340096a0 tx=0x7f7e34006b40).stop 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 msgr2=0x7f7e441053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e441053b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7e38013510 tx=0x7f7e3800b040).stop 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 shutdown_connections 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7e2c02ff60 0x7f7e2c032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 --2- 172.21.15.170:0/4203570394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e440f51a0 0x7f7e441053b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 >> 172.21.15.170:0/4203570394 conn(0x7f7e440f09a0 msgr2=0x7f7e440f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 shutdown_connections 2023-12-08T10:08:53.817 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:53.814+0000 7f7e43fff700 1 -- 172.21.15.170:0/4203570394 wait complete. 2023-12-08T10:08:53.818 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:54.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:52.327128+0000 mgr.smithi023.tgiigg (mgr.14164) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:54.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:53.814092+0000 mon.smithi023 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.170:0/4203570394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:55.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:55.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:56.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:54.327402+0000 mgr.smithi023.tgiigg (mgr.14164) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:56.404 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 -- 172.21.15.170:0/342897339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 --2- 172.21.15.170:0/342897339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f6a60009fb0 tx=0x7f6a60009b30).stop 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 -- 172.21.15.170:0/342897339 shutdown_connections 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 --2- 172.21.15.170:0/342897339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 -- 172.21.15.170:0/342897339 >> 172.21.15.170:0/342897339 conn(0x7f6a700f0b70 msgr2=0x7f6a700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 -- 172.21.15.170:0/342897339 shutdown_connections 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.401+0000 7f6a74c3c700 1 -- 172.21.15.170:0/342897339 wait complete. 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a74c3c700 1 Processor -- start 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a74c3c700 1 -- start start 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a74c3c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a74c3c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700fb820 con 0x7f6a700f5410 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a6e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a6e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33964/0 (socket says 172.21.15.170:33964) 2023-12-08T10:08:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.402+0000 7f6a6e59c700 1 -- 172.21.15.170:0/1056659343 learned_addr learned my addr 172.21.15.170:0/1056659343 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:56.406 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a6e59c700 1 -- 172.21.15.170:0/1056659343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60005040 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a6e59c700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6a60013510 tx=0x7f6a6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a60009320 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a60013f00 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a700fb9c0 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a60014d90 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.403+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a700fbe50 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.404+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a600095f0 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.404+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70059100 con 0x7f6a700f5410 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.404+0000 7f6a5effd700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6a5802ff60 0x7f6a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:56.407 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.404+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a600410c0 con 0x7f6a700f5410 2023-12-08T10:08:56.408 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.404+0000 7f6a6dd9b700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6a5802ff60 0x7f6a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:56.408 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.405+0000 7f6a6dd9b700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6a5802ff60 0x7f6a58032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6a640096a0 tx=0x7f6a64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:56.411 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.409+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a60014420 con 0x7f6a700f5410 2023-12-08T10:08:56.620 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.615+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a700fae60 con 0x7f6a700f5410 2023-12-08T10:08:56.621 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.616+0000 7f6a5effd700 1 -- 172.21.15.170:0/1056659343 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a6001d7f0 con 0x7f6a700f5410 2023-12-08T10:08:56.621 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:56.621 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6a5802ff60 msgr2=0x7f6a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6a5802ff60 0x7f6a58032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f6a640096a0 tx=0x7f6a64006b40).stop 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a700fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f6a60013510 tx=0x7f6a6000b040).stop 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 shutdown_connections 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6a5802ff60 0x7f6a58032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 --2- 172.21.15.170:0/1056659343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 >> 172.21.15.170:0/1056659343 conn(0x7f6a700f0b70 msgr2=0x7f6a7005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 shutdown_connections 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:56.618+0000 7f6a74c3c700 1 -- 172.21.15.170:0/1056659343 wait complete. 2023-12-08T10:08:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:08:57.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:56 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:56.617712+0000 mon.smithi023 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.170:0/1056659343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:08:58.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:08:58.009 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:08:58.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:56.327740+0000 mgr.smithi023.tgiigg (mgr.14164) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:08:59.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 -- 172.21.15.170:0/1000544048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 msgr2=0x7f74700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:59.137 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 --2- 172.21.15.170:0/1000544048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74700f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f7464009fb0 tx=0x7f7464009b30).stop 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 -- 172.21.15.170:0/1000544048 shutdown_connections 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 --2- 172.21.15.170:0/1000544048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74700f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 -- 172.21.15.170:0/1000544048 >> 172.21.15.170:0/1000544048 conn(0x7f74700f09a0 msgr2=0x7f74700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 -- 172.21.15.170:0/1000544048 shutdown_connections 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 -- 172.21.15.170:0/1000544048 wait complete. 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.134+0000 7f747535d700 1 Processor -- start 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f747535d700 1 -- start start 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f747535d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74701064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f747535d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74701069d0 con 0x7f74700f51a0 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f746ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74701064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f746ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74701064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33978/0 (socket says 172.21.15.170:33978) 2023-12-08T10:08:59.138 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f746ffff700 1 -- 172.21.15.170:0/388454637 learned_addr learned my addr 172.21.15.170:0/388454637 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.135+0000 7f746ffff700 1 -- 172.21.15.170:0/388454637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7464005040 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.136+0000 7f746ffff700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74701064e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7464009830 tx=0x7f746400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.136+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7464009320 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.136+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7470106bd0 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.136+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74640136c0 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.136+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f746401ee90 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.136+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7470107060 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.137+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f746401e420 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.137+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7470045ad0 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.137+0000 7f746cff9700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f745802ff10 0x7f74580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.137+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7464040f40 con 0x7f74700f51a0 2023-12-08T10:08:59.140 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.137+0000 7f746f7fe700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f745802ff10 0x7f74580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:08:59.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.137+0000 7f746f7fe700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f745802ff10 0x7f74580323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f74600096a0 tx=0x7f7460006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:08:59.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.141+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f746406d2b0 con 0x7f74700f51a0 2023-12-08T10:08:59.350 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.348+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7470107f10 con 0x7f74700f51a0 2023-12-08T10:08:59.351 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.348+0000 7f746cff9700 1 -- 172.21.15.170:0/388454637 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f746402c2d0 con 0x7f74700f51a0 2023-12-08T10:08:59.351 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:08:59.351 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f745802ff10 msgr2=0x7f74580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f745802ff10 0x7f74580323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f74600096a0 tx=0x7f7460006b40).stop 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 msgr2=0x7f74701064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74701064e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f7464009830 tx=0x7f746400b040).stop 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 shutdown_connections 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f745802ff10 0x7f74580323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 --2- 172.21.15.170:0/388454637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700f51a0 0x7f74701064e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 >> 172.21.15.170:0/388454637 conn(0x7f74700f09a0 msgr2=0x7f747005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 shutdown_connections 2023-12-08T10:08:59.353 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:08:59.350+0000 7f747535d700 1 -- 172.21.15.170:0/388454637 wait complete. 2023-12-08T10:08:59.354 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:00.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:08:58.328001+0000 mgr.smithi023.tgiigg (mgr.14164) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:00.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:08:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:08:59.350165+0000 mon.smithi023 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.170:0/388454637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:00.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:00.751 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:01.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- 172.21.15.170:0/3392804674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 msgr2=0x7f54480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 --2- 172.21.15.170:0/3392804674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54480f55c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f5444009fb0 tx=0x7f5444009b30).stop 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- 172.21.15.170:0/3392804674 shutdown_connections 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 --2- 172.21.15.170:0/3392804674 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54480f55c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- 172.21.15.170:0/3392804674 >> 172.21.15.170:0/3392804674 conn(0x7f54480f09a0 msgr2=0x7f54480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- 172.21.15.170:0/3392804674 shutdown_connections 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- 172.21.15.170:0/3392804674 wait complete. 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 Processor -- start 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- start start 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54481053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:01.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.908+0000 7f544ec16700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54481058a0 con 0x7f54480f51a0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544dc14700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54481053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544dc14700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54481053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33988/0 (socket says 172.21.15.170:33988) 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544dc14700 1 -- 172.21.15.170:0/3383937535 learned_addr learned my addr 172.21.15.170:0/3383937535 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544dc14700 1 -- 172.21.15.170:0/3383937535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5444005040 con 0x7f54480f51a0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544dc14700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54481053b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5444009830 tx=0x7f544400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5444009320 con 0x7f54480f51a0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5448105aa0 con 0x7f54480f51a0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54440136c0 con 0x7f54480f51a0 2023-12-08T10:09:01.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f544401ee90 con 0x7f54480f51a0 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.909+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5448105f30 con 0x7f54480f51a0 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.910+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f544401e420 con 0x7f54480f51a0 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.910+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5448045ad0 con 0x7f54480f51a0 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.910+0000 7f543e7fc700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f543402ff10 0x7f54340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.910+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5444040ee0 con 0x7f54480f51a0 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.911+0000 7f544d413700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f543402ff10 0x7f54340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:01.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.911+0000 7f544d413700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f543402ff10 0x7f54340323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f54380096a0 tx=0x7f5438006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:01.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:01.916+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54440095f0 con 0x7f54480f51a0 2023-12-08T10:09:02.124 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.122+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5448106de0 con 0x7f54480f51a0 2023-12-08T10:09:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.122+0000 7f543e7fc700 1 -- 172.21.15.170:0/3383937535 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f544402c2d0 con 0x7f54480f51a0 2023-12-08T10:09:02.125 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:02.125 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:02.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f543402ff10 msgr2=0x7f54340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:02.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f543402ff10 0x7f54340323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f54380096a0 tx=0x7f5438006b40).stop 2023-12-08T10:09:02.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 msgr2=0x7f54481053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54481053b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f5444009830 tx=0x7f544400b040).stop 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 shutdown_connections 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f543402ff10 0x7f54340323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 --2- 172.21.15.170:0/3383937535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54480f51a0 0x7f54481053b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 >> 172.21.15.170:0/3383937535 conn(0x7f54480f09a0 msgr2=0x7f54480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 shutdown_connections 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:02.125+0000 7f544ec16700 1 -- 172.21.15.170:0/3383937535 wait complete. 2023-12-08T10:09:02.127 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:02.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:00.328389+0000 mgr.smithi023.tgiigg (mgr.14164) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:03.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:02.124165+0000 mon.smithi023 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.170:0/3383937535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:03.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:03.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:04.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:02.328768+0000 mgr.smithi023.tgiigg (mgr.14164) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 -- 172.21.15.170:0/3396062752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 msgr2=0x7f77b40f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 --2- 172.21.15.170:0/3396062752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b40f6df0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f77a4009fb0 tx=0x7f77a4009b30).stop 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 -- 172.21.15.170:0/3396062752 shutdown_connections 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 --2- 172.21.15.170:0/3396062752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b40f6df0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 -- 172.21.15.170:0/3396062752 >> 172.21.15.170:0/3396062752 conn(0x7f77b40f02e0 msgr2=0x7f77b40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 -- 172.21.15.170:0/3396062752 shutdown_connections 2023-12-08T10:09:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.701+0000 7f77b9571700 1 -- 172.21.15.170:0/3396062752 wait complete. 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.702+0000 7f77b9571700 1 Processor -- start 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.702+0000 7f77b9571700 1 -- start start 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.702+0000 7f77b9571700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b4109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.702+0000 7f77b9571700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b40f52b0 con 0x7f77b40f69d0 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b4109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b4109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:51196/0 (socket says 172.21.15.170:51196) 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b2ffd700 1 -- 172.21.15.170:0/1830187267 learned_addr learned my addr 172.21.15.170:0/1830187267 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b2ffd700 1 -- 172.21.15.170:0/1830187267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a4005040 con 0x7f77b40f69d0 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b2ffd700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b4109360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f77a4013510 tx=0x7f77a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77a4009320 con 0x7f77b40f69d0 2023-12-08T10:09:04.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b40f54b0 con 0x7f77b40f69d0 2023-12-08T10:09:04.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77a4013f00 con 0x7f77b40f69d0 2023-12-08T10:09:04.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77a4014d90 con 0x7f77b40f69d0 2023-12-08T10:09:04.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.703+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b4109a90 con 0x7f77b40f69d0 2023-12-08T10:09:04.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.704+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77a40095f0 con 0x7f77b40f69d0 2023-12-08T10:09:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.704+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b4045ad0 con 0x7f77b40f69d0 2023-12-08T10:09:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.704+0000 7f779b7fe700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f779c02ff60 0x7f779c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.704+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77a4040fe0 con 0x7f77b40f69d0 2023-12-08T10:09:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.705+0000 7f77b27fc700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f779c02ff60 0x7f779c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:04.707 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.705+0000 7f77b27fc700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f779c02ff60 0x7f779c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f77a80096a0 tx=0x7f77a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:04.710 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.709+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77a401d7f0 con 0x7f77b40f69d0 2023-12-08T10:09:04.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.912+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77b40fae50 con 0x7f77b40f69d0 2023-12-08T10:09:04.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.913+0000 7f779b7fe700 1 -- 172.21.15.170:0/1830187267 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77a4010230 con 0x7f77b40f69d0 2023-12-08T10:09:04.914 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:04.915 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f779c02ff60 msgr2=0x7f779c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f779c02ff60 0x7f779c032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f77a80096a0 tx=0x7f77a8006b40).stop 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 msgr2=0x7f77b4109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b4109360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f77a4013510 tx=0x7f77a400b040).stop 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 shutdown_connections 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f779c02ff60 0x7f779c032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 --2- 172.21.15.170:0/1830187267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77b40f69d0 0x7f77b4109360 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:04.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 >> 172.21.15.170:0/1830187267 conn(0x7f77b40f02e0 msgr2=0x7f77b40f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:04.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 shutdown_connections 2023-12-08T10:09:04.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:04.915+0000 7f77b9571700 1 -- 172.21.15.170:0/1830187267 wait complete. 2023-12-08T10:09:04.917 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:06.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:04.329149+0000 mgr.smithi023.tgiigg (mgr.14164) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:06.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:04.913854+0000 mon.smithi023 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.170:0/1830187267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:06.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:06.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:07.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 -- 172.21.15.170:0/2014394784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 msgr2=0x7ff3c80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:07.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 --2- 172.21.15.170:0/2014394784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c80f4be0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff3bc009fb0 tx=0x7ff3bc009b30).stop 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 -- 172.21.15.170:0/2014394784 shutdown_connections 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 --2- 172.21.15.170:0/2014394784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c80f4be0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 -- 172.21.15.170:0/2014394784 >> 172.21.15.170:0/2014394784 conn(0x7ff3c80f02e0 msgr2=0x7ff3c80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 -- 172.21.15.170:0/2014394784 shutdown_connections 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 -- 172.21.15.170:0/2014394784 wait complete. 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 Processor -- start 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.500+0000 7ff3cc816700 1 -- start start 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3cc816700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3cc816700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8106bf0 con 0x7ff3c80f47c0 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3c659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3c659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:51214/0 (socket says 172.21.15.170:51214) 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3c659c700 1 -- 172.21.15.170:0/1128499221 learned_addr learned my addr 172.21.15.170:0/1128499221 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3c659c700 1 -- 172.21.15.170:0/1128499221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc005040 con 0x7ff3c80f47c0 2023-12-08T10:09:07.503 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.501+0000 7ff3c659c700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c8106700 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff3bc014000 tx=0x7ff3bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3bc0095f0 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c8106df0 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3bc007e70 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c8107280 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3bc014ea0 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3bc025070 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.502+0000 7ff3b6ffd700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3b002ffb0 0x7ff3b0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.503+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3bc040940 con 0x7ff3c80f47c0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.503+0000 7ff3c5d9b700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3b002ffb0 0x7ff3b0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:07.505 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.503+0000 7ff3c5d9b700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3b002ffb0 0x7ff3b0032470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff3b80096a0 tx=0x7ff3b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:07.506 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.504+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac004fa0 con 0x7ff3c80f47c0 2023-12-08T10:09:07.510 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.509+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3bc026350 con 0x7ff3c80f47c0 2023-12-08T10:09:07.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.713+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3ac005670 con 0x7ff3c80f47c0 2023-12-08T10:09:07.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.714+0000 7ff3b6ffd700 1 -- 172.21.15.170:0/1128499221 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff3bc012330 con 0x7ff3c80f47c0 2023-12-08T10:09:07.715 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:07.715 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:07.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3b002ffb0 msgr2=0x7ff3b0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:07.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3b002ffb0 0x7ff3b0032470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7ff3b80096a0 tx=0x7ff3b8006b40).stop 2023-12-08T10:09:07.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 msgr2=0x7ff3c8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:07.716 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c8106700 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff3bc014000 tx=0x7ff3bc00b040).stop 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 shutdown_connections 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff3b002ffb0 0x7ff3b0032470 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 --2- 172.21.15.170:0/1128499221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3c80f47c0 0x7ff3c8106700 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 >> 172.21.15.170:0/1128499221 conn(0x7ff3c80f02e0 msgr2=0x7ff3c80f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 shutdown_connections 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:07.715+0000 7ff3cc816700 1 -- 172.21.15.170:0/1128499221 wait complete. 2023-12-08T10:09:07.717 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:08.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:06.329513+0000 mgr.smithi023.tgiigg (mgr.14164) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:08.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:07.714234+0000 mon.smithi023 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.170:0/1128499221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:09.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:09.115 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:10.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:08.329904+0000 mgr.smithi023.tgiigg (mgr.14164) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:10.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 -- 172.21.15.170:0/2580847393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 msgr2=0x7f399c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:10.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 --2- 172.21.15.170:0/2580847393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c0f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f398c009fb0 tx=0x7f398c009b30).stop 2023-12-08T10:09:10.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 -- 172.21.15.170:0/2580847393 shutdown_connections 2023-12-08T10:09:10.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 --2- 172.21.15.170:0/2580847393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c0f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:10.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 -- 172.21.15.170:0/2580847393 >> 172.21.15.170:0/2580847393 conn(0x7f399c0f1c70 msgr2=0x7f399c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:10.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 -- 172.21.15.170:0/2580847393 shutdown_connections 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.324+0000 7f39a46d2700 1 -- 172.21.15.170:0/2580847393 wait complete. 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.325+0000 7f39a46d2700 1 Processor -- start 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.325+0000 7f39a46d2700 1 -- start start 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.325+0000 7f39a46d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.325+0000 7f39a46d2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399c106bc0 con 0x7f399c0f6510 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a246e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a246e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:51218/0 (socket says 172.21.15.170:51218) 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a246e700 1 -- 172.21.15.170:0/2937592967 learned_addr learned my addr 172.21.15.170:0/2937592967 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a246e700 1 -- 172.21.15.170:0/2937592967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c005040 con 0x7f399c0f6510 2023-12-08T10:09:10.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a246e700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c1066d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f398c013510 tx=0x7f398c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f398c009320 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399c106dc0 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f398c013f00 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f398c014d90 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.326+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f399c107250 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.327+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f398c0095f0 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.327+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399c045ad0 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.327+0000 7f3992ffd700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f398802ff60 0x7f3988032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.327+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f398c0410c0 con 0x7f399c0f6510 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.327+0000 7f39a1c6d700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f398802ff60 0x7f3988032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:10.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.328+0000 7f39a1c6d700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f398802ff60 0x7f3988032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f39980096a0 tx=0x7f3998006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:10.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.332+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f398c014420 con 0x7f399c0f6510 2023-12-08T10:09:10.536 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.535+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f399c108100 con 0x7f399c0f6510 2023-12-08T10:09:10.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.536+0000 7f3992ffd700 1 -- 172.21.15.170:0/2937592967 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f398c01d7f0 con 0x7f399c0f6510 2023-12-08T10:09:10.537 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:10.537 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f398802ff60 msgr2=0x7f3988032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f398802ff60 0x7f3988032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f39980096a0 tx=0x7f3998006b40).stop 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 msgr2=0x7f399c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c1066d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f398c013510 tx=0x7f398c00b040).stop 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 shutdown_connections 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f398802ff60 0x7f3988032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 --2- 172.21.15.170:0/2937592967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f399c0f6510 0x7f399c1066d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:10.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 >> 172.21.15.170:0/2937592967 conn(0x7f399c0f1c70 msgr2=0x7f399c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:10.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 shutdown_connections 2023-12-08T10:09:10.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:10.537+0000 7f39a46d2700 1 -- 172.21.15.170:0/2937592967 wait complete. 2023-12-08T10:09:10.540 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:11.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:10.536450+0000 mon.smithi023 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.170:0/2937592967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:11.913 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:11.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:12.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:10.330164+0000 mgr.smithi023.tgiigg (mgr.14164) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 -- 172.21.15.170:0/2831683454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 msgr2=0x7f5f440fb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 --2- 172.21.15.170:0/2831683454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f440fb4f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5f34009fb0 tx=0x7f5f34009b30).stop 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 -- 172.21.15.170:0/2831683454 shutdown_connections 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 --2- 172.21.15.170:0/2831683454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f440fb4f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 -- 172.21.15.170:0/2831683454 >> 172.21.15.170:0/2831683454 conn(0x7f5f440f0b70 msgr2=0x7f5f440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 -- 172.21.15.170:0/2831683454 shutdown_connections 2023-12-08T10:09:13.091 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 -- 172.21.15.170:0/2831683454 wait complete. 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.088+0000 7f5f49edf700 1 Processor -- start 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f49edf700 1 -- start start 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f49edf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f441088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f49edf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f44108dd0 con 0x7f5f44060280 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f437fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f441088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f437fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f441088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59596/0 (socket says 172.21.15.170:59596) 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f437fe700 1 -- 172.21.15.170:0/2844937592 learned_addr learned my addr 172.21.15.170:0/2844937592 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f437fe700 1 -- 172.21.15.170:0/2844937592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34005040 con 0x7f5f44060280 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.089+0000 7f5f437fe700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f441088e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5f34009830 tx=0x7f5f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f34009320 con 0x7f5f44060280 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f44108fd0 con 0x7f5f44060280 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f340136c0 con 0x7f5f44060280 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f3401ee90 con 0x7f5f44060280 2023-12-08T10:09:13.092 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f44109460 con 0x7f5f44060280 2023-12-08T10:09:13.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f34014440 con 0x7f5f44060280 2023-12-08T10:09:13.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.090+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f44045ad0 con 0x7f5f44060280 2023-12-08T10:09:13.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.091+0000 7f5f2bfff700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5f2c02ff10 0x7f5f2c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:13.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.091+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f34041090 con 0x7f5f44060280 2023-12-08T10:09:13.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.091+0000 7f5f42ffd700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5f2c02ff10 0x7f5f2c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:13.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.091+0000 7f5f42ffd700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5f2c02ff10 0x7f5f2c0323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5f380096a0 tx=0x7f5f38006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:13.097 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.096+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f340095f0 con 0x7f5f44060280 2023-12-08T10:09:13.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.301+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f4410a310 con 0x7f5f44060280 2023-12-08T10:09:13.303 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.302+0000 7f5f2bfff700 1 -- 172.21.15.170:0/2844937592 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5f3402c2d0 con 0x7f5f44060280 2023-12-08T10:09:13.303 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:13.303 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:13.304 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5f2c02ff10 msgr2=0x7f5f2c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5f2c02ff10 0x7f5f2c0323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f5f380096a0 tx=0x7f5f38006b40).stop 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 msgr2=0x7f5f441088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f441088e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f5f34009830 tx=0x7f5f3400b040).stop 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 shutdown_connections 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5f2c02ff10 0x7f5f2c0323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 --2- 172.21.15.170:0/2844937592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f44060280 0x7f5f441088e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 >> 172.21.15.170:0/2844937592 conn(0x7f5f440f0b70 msgr2=0x7f5f440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 shutdown_connections 2023-12-08T10:09:13.305 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:13.303+0000 7f5f49edf700 1 -- 172.21.15.170:0/2844937592 wait complete. 2023-12-08T10:09:13.306 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:12.330440+0000 mgr.smithi023.tgiigg (mgr.14164) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:13.302459+0000 mon.smithi023 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.170:0/2844937592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:14.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:14.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 -- 172.21.15.170:0/1593208003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 msgr2=0x7f112c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 --2- 172.21.15.170:0/1593208003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c0f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1114009fb0 tx=0x7f1114009b30).stop 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 -- 172.21.15.170:0/1593208003 shutdown_connections 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 --2- 172.21.15.170:0/1593208003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c0f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 -- 172.21.15.170:0/1593208003 >> 172.21.15.170:0/1593208003 conn(0x7f112c0f1c70 msgr2=0x7f112c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 -- 172.21.15.170:0/1593208003 shutdown_connections 2023-12-08T10:09:15.821 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.819+0000 7f11311ba700 1 -- 172.21.15.170:0/1593208003 wait complete. 2023-12-08T10:09:15.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.820+0000 7f11311ba700 1 Processor -- start 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.820+0000 7f11311ba700 1 -- start start 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.820+0000 7f11311ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.820+0000 7f11311ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c106c20 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f112ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f112ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59618/0 (socket says 172.21.15.170:59618) 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f112ad9d700 1 -- 172.21.15.170:0/3452404835 learned_addr learned my addr 172.21.15.170:0/3452404835 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f112ad9d700 1 -- 172.21.15.170:0/3452404835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1114005040 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f112ad9d700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1114013510 tx=0x7f111400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1114009320 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c106e20 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1114013f00 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1114014d90 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.821+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c1072b0 con 0x7f112c0f6510 2023-12-08T10:09:15.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.822+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11140263a0 con 0x7f112c0f6510 2023-12-08T10:09:15.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.822+0000 7f11237fe700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f111802ffb0 0x7f1118032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:15.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.822+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11140410b0 con 0x7f112c0f6510 2023-12-08T10:09:15.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.822+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112c045ad0 con 0x7f112c0f6510 2023-12-08T10:09:15.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.822+0000 7f112a59c700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f111802ffb0 0x7f1118032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:15.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.823+0000 7f112a59c700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f111802ffb0 0x7f1118032470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f111c0096a0 tx=0x7f111c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:15.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:15.827+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1114012920 con 0x7f112c0f6510 2023-12-08T10:09:16.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.034+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f112c107ff0 con 0x7f112c0f6510 2023-12-08T10:09:16.036 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.035+0000 7f11237fe700 1 -- 172.21.15.170:0/3452404835 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f111402b020 con 0x7f112c0f6510 2023-12-08T10:09:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:16.036 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:16.037 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f111802ffb0 msgr2=0x7f1118032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f111802ffb0 0x7f1118032470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f111c0096a0 tx=0x7f111c006b40).stop 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 msgr2=0x7f112c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1114013510 tx=0x7f111400b040).stop 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 shutdown_connections 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f111802ffb0 0x7f1118032470 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 --2- 172.21.15.170:0/3452404835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f112c0f6510 0x7f112c106730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 >> 172.21.15.170:0/3452404835 conn(0x7f112c0f1c70 msgr2=0x7f112c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 shutdown_connections 2023-12-08T10:09:16.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:16.036+0000 7f11311ba700 1 -- 172.21.15.170:0/3452404835 wait complete. 2023-12-08T10:09:16.039 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:14.330725+0000 mgr.smithi023.tgiigg (mgr.14164) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:17.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:16.035536+0000 mon.smithi023 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.170:0/3452404835' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:17.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:17.428 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:16.331057+0000 mgr.smithi023.tgiigg (mgr.14164) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:18.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.608+0000 7ff99899c700 1 -- 172.21.15.170:0/714921764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 msgr2=0x7ff9940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.608+0000 7ff99899c700 1 --2- 172.21.15.170:0/714921764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9940f55c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ff984009fb0 tx=0x7ff984009b30).stop 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 -- 172.21.15.170:0/714921764 shutdown_connections 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 --2- 172.21.15.170:0/714921764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9940f55c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 -- 172.21.15.170:0/714921764 >> 172.21.15.170:0/714921764 conn(0x7ff9940f09a0 msgr2=0x7ff9940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 -- 172.21.15.170:0/714921764 shutdown_connections 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 -- 172.21.15.170:0/714921764 wait complete. 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 Processor -- start 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 -- start start 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9941091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:18.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.609+0000 7ff99899c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff994060220 con 0x7ff9940f51a0 2023-12-08T10:09:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff9937fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9941091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff9937fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9941091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59648/0 (socket says 172.21.15.170:59648) 2023-12-08T10:09:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff9937fe700 1 -- 172.21.15.170:0/1212567073 learned_addr learned my addr 172.21.15.170:0/1212567073 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff9937fe700 1 -- 172.21.15.170:0/1212567073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff984005040 con 0x7ff9940f51a0 2023-12-08T10:09:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff9937fe700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9941091d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff984013510 tx=0x7ff98400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff984009320 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff994060420 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.610+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff984013f00 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.611+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff984014dc0 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.611+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff994109880 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.611+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9840095f0 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.611+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff994059100 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.611+0000 7ff97bfff700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff97c02ff60 0x7ff97c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.611+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9840410f0 con 0x7ff9940f51a0 2023-12-08T10:09:18.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.612+0000 7ff992ffd700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff97c02ff60 0x7ff97c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:18.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.612+0000 7ff992ffd700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff97c02ff60 0x7ff97c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff9880096a0 tx=0x7ff988006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:18.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.616+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff984014420 con 0x7ff9940f51a0 2023-12-08T10:09:18.822 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.820+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff994045ad0 con 0x7ff9940f51a0 2023-12-08T10:09:18.823 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.821+0000 7ff97bfff700 1 -- 172.21.15.170:0/1212567073 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff984010230 con 0x7ff9940f51a0 2023-12-08T10:09:18.823 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:18.823 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:18.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff97c02ff60 msgr2=0x7ff97c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:18.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff97c02ff60 0x7ff97c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff9880096a0 tx=0x7ff988006b40).stop 2023-12-08T10:09:18.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 msgr2=0x7ff9941091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:18.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9941091d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7ff984013510 tx=0x7ff98400b040).stop 2023-12-08T10:09:18.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 shutdown_connections 2023-12-08T10:09:18.826 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff97c02ff60 0x7ff97c032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:18.826 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 --2- 172.21.15.170:0/1212567073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9940f51a0 0x7ff9941091d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:18.826 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 >> 172.21.15.170:0/1212567073 conn(0x7ff9940f09a0 msgr2=0x7ff9940f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:18.826 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 shutdown_connections 2023-12-08T10:09:18.826 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:18.823+0000 7ff99899c700 1 -- 172.21.15.170:0/1212567073 wait complete. 2023-12-08T10:09:18.826 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:18.821737+0000 mon.smithi023 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.170:0/1212567073' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:20.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:18.331418+0000 mgr.smithi023.tgiigg (mgr.14164) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:20.217 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:20.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:21.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.380+0000 7fac172da700 1 -- 172.21.15.170:0/3537806162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 msgr2=0x7fac100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:21.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.380+0000 7fac172da700 1 --2- 172.21.15.170:0/3537806162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac100f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fac00009fb0 tx=0x7fac00009b30).stop 2023-12-08T10:09:21.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 -- 172.21.15.170:0/3537806162 shutdown_connections 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 --2- 172.21.15.170:0/3537806162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac100f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 -- 172.21.15.170:0/3537806162 >> 172.21.15.170:0/3537806162 conn(0x7fac100f09a0 msgr2=0x7fac100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 -- 172.21.15.170:0/3537806162 shutdown_connections 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 -- 172.21.15.170:0/3537806162 wait complete. 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 Processor -- start 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 -- start start 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac101053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac172da700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac101058a0 con 0x7fac100f51a0 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac162d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac101053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.381+0000 7fac162d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac101053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59658/0 (socket says 172.21.15.170:59658) 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac162d8700 1 -- 172.21.15.170:0/1791688667 learned_addr learned my addr 172.21.15.170:0/1791688667 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:21.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac162d8700 1 -- 172.21.15.170:0/1791688667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac00005040 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac162d8700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac101053b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fac00013510 tx=0x7fac0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac00009320 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac10105aa0 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac00013f00 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac00014dc0 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.382+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac10105f30 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.383+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac0001d3d0 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.383+0000 7fac06ffd700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.383+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac10045ad0 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.383+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac00041320 con 0x7fac100f51a0 2023-12-08T10:09:21.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.383+0000 7fac15ad7700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:21.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.384+0000 7fac15ad7700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fabfc02ff60 0x7fabfc032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fac0c0096a0 tx=0x7fac0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:21.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.388+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac00012900 con 0x7fac100f51a0 2023-12-08T10:09:21.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.595+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac10106de0 con 0x7fac100f51a0 2023-12-08T10:09:21.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.596+0000 7fac06ffd700 1 -- 172.21.15.170:0/1791688667 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac0002b030 con 0x7fac100f51a0 2023-12-08T10:09:21.597 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:21.597 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fabfc02ff60 msgr2=0x7fabfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fabfc02ff60 0x7fabfc032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fac0c0096a0 tx=0x7fac0c006b40).stop 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 msgr2=0x7fac101053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac101053b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fac00013510 tx=0x7fac0000b040).stop 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 shutdown_connections 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 --2- 172.21.15.170:0/1791688667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac100f51a0 0x7fac101053b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 >> 172.21.15.170:0/1791688667 conn(0x7fac100f09a0 msgr2=0x7fac100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:21.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.597+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 shutdown_connections 2023-12-08T10:09:21.600 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:21.598+0000 7fac172da700 1 -- 172.21.15.170:0/1791688667 wait complete. 2023-12-08T10:09:21.600 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:20.331812+0000 mgr.smithi023.tgiigg (mgr.14164) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:21.596459+0000 mon.smithi023 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.170:0/1791688667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:22.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:22.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:22.451311+0000 mon.smithi023 (mon.0) 176 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:09:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:22.453999+0000 mon.smithi023 (mon.0) 177 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 -- 172.21.15.170:0/3970089881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 msgr2=0x7f275c0f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 --2- 172.21.15.170:0/3970089881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c0f5750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f274c009fb0 tx=0x7f274c009b30).stop 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 -- 172.21.15.170:0/3970089881 shutdown_connections 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 --2- 172.21.15.170:0/3970089881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c0f5750 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 -- 172.21.15.170:0/3970089881 >> 172.21.15.170:0/3970089881 conn(0x7f275c0f0a90 msgr2=0x7f275c0f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 -- 172.21.15.170:0/3970089881 shutdown_connections 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.144+0000 7f2762869700 1 -- 172.21.15.170:0/3970089881 wait complete. 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f2762869700 1 Processor -- start 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f2762869700 1 -- start start 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f2762869700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c1058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:24.148 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f2762869700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c105dc0 con 0x7f275c0f5330 2023-12-08T10:09:24.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f275bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c1058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:24.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f275bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c1058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39082/0 (socket says 172.21.15.170:39082) 2023-12-08T10:09:24.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.145+0000 7f275bfff700 1 -- 172.21.15.170:0/782945874 learned_addr learned my addr 172.21.15.170:0/782945874 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:24.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f275bfff700 1 -- 172.21.15.170:0/782945874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c005040 con 0x7f275c0f5330 2023-12-08T10:09:24.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f275bfff700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c1058d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f274c014000 tx=0x7f274c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:24.151 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f274c0095f0 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c105fc0 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f274c007e70 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f274c014ea0 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.146+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275c106450 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.147+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f274c025070 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.147+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275c045ad0 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.147+0000 7f2758ff9700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f274402ff60 0x7f2744032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.147+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f274c040a00 con 0x7f275c0f5330 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.147+0000 7f275b7fe700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f274402ff60 0x7f2744032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:24.152 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.148+0000 7f275b7fe700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f274402ff60 0x7f2744032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f27500096a0 tx=0x7f2750006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:24.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.152+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f274c0079c0 con 0x7f275c0f5330 2023-12-08T10:09:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:22.332158+0000 mgr.smithi023.tgiigg (mgr.14164) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:24.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.358+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f275c05ea50 con 0x7f275c0f5330 2023-12-08T10:09:24.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.358+0000 7f2758ff9700 1 -- 172.21.15.170:0/782945874 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f274c00f300 con 0x7f275c0f5330 2023-12-08T10:09:24.361 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:24.361 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:24.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f274402ff60 msgr2=0x7f2744032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:24.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f274402ff60 0x7f2744032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f27500096a0 tx=0x7f2750006b40).stop 2023-12-08T10:09:24.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 msgr2=0x7f275c1058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:24.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c1058d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f274c014000 tx=0x7f274c00b040).stop 2023-12-08T10:09:24.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 shutdown_connections 2023-12-08T10:09:24.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f274402ff60 0x7f2744032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:24.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 --2- 172.21.15.170:0/782945874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0f5330 0x7f275c1058d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:24.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 >> 172.21.15.170:0/782945874 conn(0x7f275c0f0a90 msgr2=0x7f275c0f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:24.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 shutdown_connections 2023-12-08T10:09:24.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:24.360+0000 7f2762869700 1 -- 172.21.15.170:0/782945874 wait complete. 2023-12-08T10:09:24.363 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:25.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:24.360010+0000 mon.smithi023 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.170:0/782945874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:25.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:25.754 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:26.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:24.332553+0000 mgr.smithi023.tgiigg (mgr.14164) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:26.982 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 -- 172.21.15.170:0/2403012341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 msgr2=0x7f3e240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:26.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 --2- 172.21.15.170:0/2403012341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e240f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f3e14009fb0 tx=0x7f3e14009b30).stop 2023-12-08T10:09:26.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 -- 172.21.15.170:0/2403012341 shutdown_connections 2023-12-08T10:09:26.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 --2- 172.21.15.170:0/2403012341 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e240f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:26.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 -- 172.21.15.170:0/2403012341 >> 172.21.15.170:0/2403012341 conn(0x7f3e240f1c70 msgr2=0x7f3e240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:26.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 -- 172.21.15.170:0/2403012341 shutdown_connections 2023-12-08T10:09:26.983 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.980+0000 7f3e2a777700 1 -- 172.21.15.170:0/2403012341 wait complete. 2023-12-08T10:09:26.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e2a777700 1 Processor -- start 2023-12-08T10:09:26.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e2a777700 1 -- start start 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e2a777700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e2a777700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24106c20 con 0x7f3e240f6510 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e23fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e23fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39102/0 (socket says 172.21.15.170:39102) 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.981+0000 7f3e23fff700 1 -- 172.21.15.170:0/3396184293 learned_addr learned my addr 172.21.15.170:0/3396184293 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e23fff700 1 -- 172.21.15.170:0/3396184293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14005040 con 0x7f3e240f6510 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e23fff700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e24106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f3e14013510 tx=0x7f3e1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e14009320 con 0x7f3e240f6510 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e24106e20 con 0x7f3e240f6510 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e14013f00 con 0x7f3e240f6510 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e14014d90 con 0x7f3e240f6510 2023-12-08T10:09:26.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.982+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e241072b0 con 0x7f3e240f6510 2023-12-08T10:09:26.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.983+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e140095f0 con 0x7f3e240f6510 2023-12-08T10:09:26.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.983+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e24045ad0 con 0x7f3e240f6510 2023-12-08T10:09:26.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.983+0000 7f3e20ff9700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3e0c02ffb0 0x7f3e0c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:26.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.983+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e14040ff0 con 0x7f3e240f6510 2023-12-08T10:09:26.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.983+0000 7f3e237fe700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3e0c02ffb0 0x7f3e0c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:26.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.984+0000 7f3e237fe700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3e0c02ffb0 0x7f3e0c032470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f3e180096a0 tx=0x7f3e18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:26.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:26.988+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e1406d2b0 con 0x7f3e240f6510 2023-12-08T10:09:27.192 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.190+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e240fbf60 con 0x7f3e240f6510 2023-12-08T10:09:27.193 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.191+0000 7f3e20ff9700 1 -- 172.21.15.170:0/3396184293 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e14014420 con 0x7f3e240f6510 2023-12-08T10:09:27.193 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:27.193 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:27.194 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3e0c02ffb0 msgr2=0x7f3e0c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3e0c02ffb0 0x7f3e0c032470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f3e180096a0 tx=0x7f3e18006b40).stop 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 msgr2=0x7f3e24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e24106730 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f3e14013510 tx=0x7f3e1400b040).stop 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 shutdown_connections 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3e0c02ffb0 0x7f3e0c032470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 --2- 172.21.15.170:0/3396184293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e240f6510 0x7f3e24106730 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 >> 172.21.15.170:0/3396184293 conn(0x7f3e240f1c70 msgr2=0x7f3e240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 shutdown_connections 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:27.192+0000 7f3e2a777700 1 -- 172.21.15.170:0/3396184293 wait complete. 2023-12-08T10:09:27.195 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:28.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:26.332809+0000 mgr.smithi023.tgiigg (mgr.14164) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:28.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:27.192363+0000 mon.smithi023 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.170:0/3396184293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:28.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:28.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:29.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 -- 172.21.15.170:0/2183883972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 msgr2=0x7faa940f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:29.738 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 --2- 172.21.15.170:0/2183883972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f90f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7faa84009fb0 tx=0x7faa84009b30).stop 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 -- 172.21.15.170:0/2183883972 shutdown_connections 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 --2- 172.21.15.170:0/2183883972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f90f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 -- 172.21.15.170:0/2183883972 >> 172.21.15.170:0/2183883972 conn(0x7faa940f0b70 msgr2=0x7faa940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 -- 172.21.15.170:0/2183883972 shutdown_connections 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.734+0000 7faa9a09f700 1 -- 172.21.15.170:0/2183883972 wait complete. 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.735+0000 7faa9a09f700 1 Processor -- start 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.735+0000 7faa9a09f700 1 -- start start 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.735+0000 7faa9a09f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.735+0000 7faa9a09f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa940f4ff0 con 0x7faa94060280 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.735+0000 7faa937fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa937fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39130/0 (socket says 172.21.15.170:39130) 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa937fe700 1 -- 172.21.15.170:0/2713308542 learned_addr learned my addr 172.21.15.170:0/2713308542 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa937fe700 1 -- 172.21.15.170:0/2713308542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84005040 con 0x7faa94060280 2023-12-08T10:09:29.739 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa937fe700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f6b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7faa84013510 tx=0x7faa8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:29.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa84009320 con 0x7faa94060280 2023-12-08T10:09:29.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa84013f00 con 0x7faa94060280 2023-12-08T10:09:29.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa940f5190 con 0x7faa94060280 2023-12-08T10:09:29.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa84014dc0 con 0x7faa94060280 2023-12-08T10:09:29.740 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.736+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa940f5620 con 0x7faa94060280 2023-12-08T10:09:29.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.737+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa8401d3d0 con 0x7faa94060280 2023-12-08T10:09:29.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.737+0000 7faa8bfff700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faa8002ff60 0x7faa80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:29.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.737+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa840410f0 con 0x7faa94060280 2023-12-08T10:09:29.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.737+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa94059100 con 0x7faa94060280 2023-12-08T10:09:29.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.737+0000 7faa92ffd700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faa8002ff60 0x7faa80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:29.741 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.738+0000 7faa92ffd700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faa8002ff60 0x7faa80032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7faa7c0096a0 tx=0x7faa7c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:29.744 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.742+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa84012900 con 0x7faa94060280 2023-12-08T10:09:29.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.947+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa94045ad0 con 0x7faa94060280 2023-12-08T10:09:29.949 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.947+0000 7faa8bfff700 1 -- 172.21.15.170:0/2713308542 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa8402b020 con 0x7faa94060280 2023-12-08T10:09:29.950 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:29.950 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faa8002ff60 msgr2=0x7faa80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faa8002ff60 0x7faa80032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7faa7c0096a0 tx=0x7faa7c006b40).stop 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 msgr2=0x7faa940f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f6b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7faa84013510 tx=0x7faa8400b040).stop 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 shutdown_connections 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7faa8002ff60 0x7faa80032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 --2- 172.21.15.170:0/2713308542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa94060280 0x7faa940f6b70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:29.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 >> 172.21.15.170:0/2713308542 conn(0x7faa940f0b70 msgr2=0x7faa940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:29.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 shutdown_connections 2023-12-08T10:09:29.952 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:29.949+0000 7faa9a09f700 1 -- 172.21.15.170:0/2713308542 wait complete. 2023-12-08T10:09:29.952 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:30.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:28.333135+0000 mgr.smithi023.tgiigg (mgr.14164) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:30.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:29.948989+0000 mon.smithi023 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.170:0/2713308542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:31.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:31.361 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:32.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:30.333421+0000 mgr.smithi023.tgiigg (mgr.14164) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.559+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2632986517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 msgr2=0x7fe5c40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.559+0000 7fe5c9e97700 1 --2- 172.21.15.170:0/2632986517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f90f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe5b4009fb0 tx=0x7fe5b4009b30).stop 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.559+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2632986517 shutdown_connections 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.559+0000 7fe5c9e97700 1 --2- 172.21.15.170:0/2632986517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f90f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.559+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2632986517 >> 172.21.15.170:0/2632986517 conn(0x7fe5c40f0b70 msgr2=0x7fe5c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.559+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2632986517 shutdown_connections 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.560+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2632986517 wait complete. 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.560+0000 7fe5c9e97700 1 Processor -- start 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.560+0000 7fe5c9e97700 1 -- start start 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c9e97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:32.563 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c9e97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40f4ff0 con 0x7fe5c4060280 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60200/0 (socket says 172.21.15.170:60200) 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c37fe700 1 -- 172.21.15.170:0/2151247521 learned_addr learned my addr 172.21.15.170:0/2151247521 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c37fe700 1 -- 172.21.15.170:0/2151247521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b4005040 con 0x7fe5c4060280 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c37fe700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f6b70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe5b4013510 tx=0x7fe5b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b4009320 con 0x7fe5c4060280 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c40f5190 con 0x7fe5c4060280 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5b4013f00 con 0x7fe5c4060280 2023-12-08T10:09:32.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5b4014dc0 con 0x7fe5c4060280 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.561+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c40f5620 con 0x7fe5c4060280 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.562+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5b40263a0 con 0x7fe5c4060280 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.562+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c4059100 con 0x7fe5c4060280 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.562+0000 7fe5abfff700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe5ac02ffb0 0x7fe5ac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.563+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5b4041fc0 con 0x7fe5c4060280 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.563+0000 7fe5c2ffd700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe5ac02ffb0 0x7fe5ac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:32.566 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.563+0000 7fe5c2ffd700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe5ac02ffb0 0x7fe5ac032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe5b80096a0 tx=0x7fe5b8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:32.569 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.567+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5b40126f0 con 0x7fe5c4060280 2023-12-08T10:09:32.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.769+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5c4045ad0 con 0x7fe5c4060280 2023-12-08T10:09:32.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.770+0000 7fe5abfff700 1 -- 172.21.15.170:0/2151247521 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe5b4012330 con 0x7fe5c4060280 2023-12-08T10:09:32.773 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:32.773 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:32.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe5ac02ffb0 msgr2=0x7fe5ac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe5ac02ffb0 0x7fe5ac032470 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe5b80096a0 tx=0x7fe5b8006b40).stop 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 msgr2=0x7fe5c40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f6b70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe5b4013510 tx=0x7fe5b400b040).stop 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 shutdown_connections 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe5ac02ffb0 0x7fe5ac032470 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 --2- 172.21.15.170:0/2151247521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5c4060280 0x7fe5c40f6b70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 >> 172.21.15.170:0/2151247521 conn(0x7fe5c40f0b70 msgr2=0x7fe5c40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 shutdown_connections 2023-12-08T10:09:32.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:32.772+0000 7fe5c9e97700 1 -- 172.21.15.170:0/2151247521 wait complete. 2023-12-08T10:09:32.776 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:33.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:32.771835+0000 mon.smithi023 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.170:0/2151247521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:34.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:34.160 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:34.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:32.333813+0000 mgr.smithi023.tgiigg (mgr.14164) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 -- 172.21.15.170:0/1076807309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 msgr2=0x7f13000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 --2- 172.21.15.170:0/1076807309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13000f90f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f12f4009fb0 tx=0x7f12f4009b30).stop 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 -- 172.21.15.170:0/1076807309 shutdown_connections 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 --2- 172.21.15.170:0/1076807309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13000f90f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 -- 172.21.15.170:0/1076807309 >> 172.21.15.170:0/1076807309 conn(0x7f13000f0b70 msgr2=0x7f13000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 -- 172.21.15.170:0/1076807309 shutdown_connections 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.307+0000 7f1306d92700 1 -- 172.21.15.170:0/1076807309 wait complete. 2023-12-08T10:09:35.310 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1306d92700 1 Processor -- start 2023-12-08T10:09:35.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1306d92700 1 -- start start 2023-12-08T10:09:35.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1306d92700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13001066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:35.311 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1306d92700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1300106bd0 con 0x7f1300060280 2023-12-08T10:09:35.312 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1304b2e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13001066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1304b2e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13001066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60218/0 (socket says 172.21.15.170:60218) 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1304b2e700 1 -- 172.21.15.170:0/269036259 learned_addr learned my addr 172.21.15.170:0/269036259 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.308+0000 7f1304b2e700 1 -- 172.21.15.170:0/269036259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f4005040 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.309+0000 7f1304b2e700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13001066e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f12f4013510 tx=0x7f12f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.309+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12f4009320 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.309+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12f4013f00 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.309+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1300106dd0 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.309+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12f4014dc0 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.309+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1300107260 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.310+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12f40095f0 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.310+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1300045ad0 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.310+0000 7f12fd7fa700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f12f002ff60 0x7f12f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.310+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12f4041000 con 0x7f1300060280 2023-12-08T10:09:35.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.310+0000 7f12fffff700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f12f002ff60 0x7f12f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:35.314 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.310+0000 7f12fffff700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f12f002ff60 0x7f12f0032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f12ec0096a0 tx=0x7f12ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:35.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.315+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12f4014420 con 0x7f1300060280 2023-12-08T10:09:35.521 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.518+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1300108110 con 0x7f1300060280 2023-12-08T10:09:35.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.519+0000 7f12fd7fa700 1 -- 172.21.15.170:0/269036259 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12f4010230 con 0x7f1300060280 2023-12-08T10:09:35.522 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:35.522 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:35.523 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f12f002ff60 msgr2=0x7f12f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f12f002ff60 0x7f12f0032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f12ec0096a0 tx=0x7f12ec006b40).stop 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 msgr2=0x7f13001066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13001066e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f12f4013510 tx=0x7f12f400b040).stop 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 shutdown_connections 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f12f002ff60 0x7f12f0032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 --2- 172.21.15.170:0/269036259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1300060280 0x7f13001066e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 >> 172.21.15.170:0/269036259 conn(0x7f13000f0b70 msgr2=0x7f13000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 shutdown_connections 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:35.521+0000 7f1306d92700 1 -- 172.21.15.170:0/269036259 wait complete. 2023-12-08T10:09:35.524 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:36.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:34.334153+0000 mgr.smithi023.tgiigg (mgr.14164) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:36.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:35.520946+0000 mon.smithi023 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.170:0/269036259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:36.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:36.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:38.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- 172.21.15.170:0/3221772844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 msgr2=0x7f3af40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:38.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 --2- 172.21.15.170:0/3221772844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af40f4be0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f3ae8009fb0 tx=0x7f3ae8009b30).stop 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- 172.21.15.170:0/3221772844 shutdown_connections 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 --2- 172.21.15.170:0/3221772844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af40f4be0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- 172.21.15.170:0/3221772844 >> 172.21.15.170:0/3221772844 conn(0x7f3af40f02e0 msgr2=0x7f3af40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- 172.21.15.170:0/3221772844 shutdown_connections 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- 172.21.15.170:0/3221772844 wait complete. 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 Processor -- start 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- start start 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.111+0000 7f3af8c76700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4106bf0 con 0x7f3af40f47c0 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60246/0 (socket says 172.21.15.170:60246) 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af259c700 1 -- 172.21.15.170:0/3212095863 learned_addr learned my addr 172.21.15.170:0/3212095863 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af259c700 1 -- 172.21.15.170:0/3212095863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8005040 con 0x7f3af40f47c0 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af259c700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af4106700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3ae8013510 tx=0x7f3ae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:38.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae8009320 con 0x7f3af40f47c0 2023-12-08T10:09:38.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af4106df0 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ae8013f00 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae8014d90 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.112+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4107280 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.113+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ae8014420 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.113+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4045ad0 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.113+0000 7f3ae2ffd700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3adc02ffb0 0x7f3adc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.113+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ae8040fd0 con 0x7f3af40f47c0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.113+0000 7f3af1d9b700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3adc02ffb0 0x7f3adc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:38.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.114+0000 7f3af1d9b700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3adc02ffb0 0x7f3adc032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3ae40096a0 tx=0x7f3ae4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:38.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.118+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae806d2b0 con 0x7f3af40f47c0 2023-12-08T10:09:38.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.325+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3af4108130 con 0x7f3af40f47c0 2023-12-08T10:09:38.328 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.326+0000 7f3ae2ffd700 1 -- 172.21.15.170:0/3212095863 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ae80095f0 con 0x7f3af40f47c0 2023-12-08T10:09:38.328 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:38.328 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.327+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3adc02ffb0 msgr2=0x7f3adc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.327+0000 7f3af8c76700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3adc02ffb0 0x7f3adc032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f3ae40096a0 tx=0x7f3ae4006b40).stop 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.327+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 msgr2=0x7f3af4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.327+0000 7f3af8c76700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af4106700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f3ae8013510 tx=0x7f3ae800b040).stop 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.327+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 shutdown_connections 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.327+0000 7f3af8c76700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f3adc02ffb0 0x7f3adc032470 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.328+0000 7f3af8c76700 1 --2- 172.21.15.170:0/3212095863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3af40f47c0 0x7f3af4106700 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.328+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 >> 172.21.15.170:0/3212095863 conn(0x7f3af40f02e0 msgr2=0x7f3af40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:38.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.328+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 shutdown_connections 2023-12-08T10:09:38.331 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:38.328+0000 7f3af8c76700 1 -- 172.21.15.170:0/3212095863 wait complete. 2023-12-08T10:09:38.331 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:38.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:36.334498+0000 mgr.smithi023.tgiigg (mgr.14164) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:39.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:38 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:38.327671+0000 mon.smithi023 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.170:0/3212095863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:39.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:39.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:40.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:38.334800+0000 mgr.smithi023.tgiigg (mgr.14164) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:40.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 -- 172.21.15.170:0/1957236486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 msgr2=0x7f4b6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:40.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 --2- 172.21.15.170:0/1957236486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c0f55c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f4b5c009fb0 tx=0x7f4b5c009b30).stop 2023-12-08T10:09:40.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 -- 172.21.15.170:0/1957236486 shutdown_connections 2023-12-08T10:09:40.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 --2- 172.21.15.170:0/1957236486 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c0f55c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:40.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 -- 172.21.15.170:0/1957236486 >> 172.21.15.170:0/1957236486 conn(0x7f4b6c0f09a0 msgr2=0x7f4b6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:40.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 -- 172.21.15.170:0/1957236486 shutdown_connections 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 -- 172.21.15.170:0/1957236486 wait complete. 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 Processor -- start 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.898+0000 7f4b6b59e700 1 -- start start 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c10b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c106fc0 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c10b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c10b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:60272/0 (socket says 172.21.15.170:60272) 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6a59c700 1 -- 172.21.15.170:0/2276013603 learned_addr learned my addr 172.21.15.170:0/2276013603 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6a59c700 1 -- 172.21.15.170:0/2276013603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c005040 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6a59c700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c10b5d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4b5c013510 tx=0x7f4b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b5c009320 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.899+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c060220 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.900+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b5c013f00 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.900+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b5c014dc0 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.900+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c10bac0 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.900+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b5c0095f0 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.900+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c045ad0 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.900+0000 7f4b5affd700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4b5402ff60 0x7f4b54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:40.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.901+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b5c0410c0 con 0x7f4b6c0f51a0 2023-12-08T10:09:40.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.901+0000 7f4b69d9b700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4b5402ff60 0x7f4b54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:40.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.901+0000 7f4b69d9b700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4b5402ff60 0x7f4b54032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4b600096a0 tx=0x7f4b60006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:40.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:40.905+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b5c014420 con 0x7f4b6c0f51a0 2023-12-08T10:09:41.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.110+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b6c10c280 con 0x7f4b6c0f51a0 2023-12-08T10:09:41.113 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.112+0000 7f4b5affd700 1 -- 172.21.15.170:0/2276013603 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b5c010230 con 0x7f4b6c0f51a0 2023-12-08T10:09:41.113 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:41.114 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:41.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4b5402ff60 msgr2=0x7f4b54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:41.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4b5402ff60 0x7f4b54032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f4b600096a0 tx=0x7f4b60006b40).stop 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 msgr2=0x7f4b6c10b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c10b5d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4b5c013510 tx=0x7f4b5c00b040).stop 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 shutdown_connections 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4b5402ff60 0x7f4b54032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 --2- 172.21.15.170:0/2276013603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b6c0f51a0 0x7f4b6c10b5d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 >> 172.21.15.170:0/2276013603 conn(0x7f4b6c0f09a0 msgr2=0x7f4b6c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 shutdown_connections 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:41.113+0000 7f4b6b59e700 1 -- 172.21.15.170:0/2276013603 wait complete. 2023-12-08T10:09:41.116 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:42.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:40.335091+0000 mgr.smithi023.tgiigg (mgr.14164) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:42.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:41.112759+0000 mon.smithi023 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.170:0/2276013603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:42.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:42.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:43.643 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.640+0000 7fc4bc638700 1 -- 172.21.15.170:0/227696434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 msgr2=0x7fc4b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 --2- 172.21.15.170:0/227696434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b40f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fc4a4009fb0 tx=0x7fc4a4009b30).stop 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 -- 172.21.15.170:0/227696434 shutdown_connections 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 --2- 172.21.15.170:0/227696434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b40f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 -- 172.21.15.170:0/227696434 >> 172.21.15.170:0/227696434 conn(0x7fc4b40f0b70 msgr2=0x7fc4b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 -- 172.21.15.170:0/227696434 shutdown_connections 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 -- 172.21.15.170:0/227696434 wait complete. 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 Processor -- start 2023-12-08T10:09:43.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.641+0000 7fc4bc638700 1 -- start start 2023-12-08T10:09:43.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.642+0000 7fc4bc638700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b4105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:43.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.642+0000 7fc4bc638700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b4105af0 con 0x7fc4b40f5410 2023-12-08T10:09:43.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.642+0000 7fc4ba3d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b4105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.642+0000 7fc4ba3d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b4105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40232/0 (socket says 172.21.15.170:40232) 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.642+0000 7fc4ba3d4700 1 -- 172.21.15.170:0/136518917 learned_addr learned my addr 172.21.15.170:0/136518917 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.642+0000 7fc4ba3d4700 1 -- 172.21.15.170:0/136518917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4005040 con 0x7fc4b40f5410 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.643+0000 7fc4ba3d4700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b4105600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc4a4013510 tx=0x7fc4a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.643+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a4009320 con 0x7fc4b40f5410 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.643+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4105cf0 con 0x7fc4b40f5410 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.643+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4a4013f00 con 0x7fc4b40f5410 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.643+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4a4014d90 con 0x7fc4b40f5410 2023-12-08T10:09:43.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.643+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b4106180 con 0x7fc4b40f5410 2023-12-08T10:09:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.644+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4a40095f0 con 0x7fc4b40f5410 2023-12-08T10:09:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.644+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4045ad0 con 0x7fc4b40f5410 2023-12-08T10:09:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.644+0000 7fc4aaffd700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc4a002ff60 0x7fc4a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.644+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4a4040fd0 con 0x7fc4b40f5410 2023-12-08T10:09:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.644+0000 7fc4b9bd3700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc4a002ff60 0x7fc4a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:43.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.645+0000 7fc4b9bd3700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc4a002ff60 0x7fc4a0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc4b00096a0 tx=0x7fc4b0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:43.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.649+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4a4014420 con 0x7fc4b40f5410 2023-12-08T10:09:43.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.855+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4b4106fc0 con 0x7fc4b40f5410 2023-12-08T10:09:43.858 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.856+0000 7fc4aaffd700 1 -- 172.21.15.170:0/136518917 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc4a401d7f0 con 0x7fc4b40f5410 2023-12-08T10:09:43.858 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:43.859 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc4a002ff60 msgr2=0x7fc4a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc4a002ff60 0x7fc4a0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc4b00096a0 tx=0x7fc4b0006b40).stop 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 msgr2=0x7fc4b4105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b4105600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fc4a4013510 tx=0x7fc4a400b040).stop 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 shutdown_connections 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc4a002ff60 0x7fc4a0032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 --2- 172.21.15.170:0/136518917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4b40f5410 0x7fc4b4105600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.857+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 >> 172.21.15.170:0/136518917 conn(0x7fc4b40f0b70 msgr2=0x7fc4b40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.858+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 shutdown_connections 2023-12-08T10:09:43.860 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:43.858+0000 7fc4bc638700 1 -- 172.21.15.170:0/136518917 wait complete. 2023-12-08T10:09:43.861 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:44.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:42.335408+0000 mgr.smithi023.tgiigg (mgr.14164) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:44.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:43.857739+0000 mon.smithi023 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.170:0/136518917' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:45.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:45.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:46.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:46 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:44.335712+0000 mgr.smithi023.tgiigg (mgr.14164) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 -- 172.21.15.170:0/2474536028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 msgr2=0x7f49300fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 --2- 172.21.15.170:0/2474536028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49300fb570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f492c009fb0 tx=0x7f492c009b30).stop 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 -- 172.21.15.170:0/2474536028 shutdown_connections 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 --2- 172.21.15.170:0/2474536028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49300fb570 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 -- 172.21.15.170:0/2474536028 >> 172.21.15.170:0/2474536028 conn(0x7f49300f0b70 msgr2=0x7f49300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 -- 172.21.15.170:0/2474536028 shutdown_connections 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.460+0000 7f4937746700 1 -- 172.21.15.170:0/2474536028 wait complete. 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f4937746700 1 Processor -- start 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f4937746700 1 -- start start 2023-12-08T10:09:46.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f4937746700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49301088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:46.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f4937746700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4930108dc0 con 0x7f4930060200 2023-12-08T10:09:46.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f49354e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49301088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f49354e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49301088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40252/0 (socket says 172.21.15.170:40252) 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f49354e2700 1 -- 172.21.15.170:0/541819416 learned_addr learned my addr 172.21.15.170:0/541819416 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.461+0000 7f49354e2700 1 -- 172.21.15.170:0/541819416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c005040 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.462+0000 7f49354e2700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49301088d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f492c013510 tx=0x7f492c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.462+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f492c009320 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.462+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4930108fc0 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.462+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f492c013f00 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.462+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f492c014dc0 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.462+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4930109450 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.463+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f492c0095f0 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.463+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4930045ad0 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.463+0000 7f4925ffb700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f491c02ff60 0x7f491c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.463+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f492c041040 con 0x7f4930060200 2023-12-08T10:09:46.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.463+0000 7f4934ce1700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f491c02ff60 0x7f491c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:46.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.464+0000 7f4934ce1700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f491c02ff60 0x7f491c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f49200096a0 tx=0x7f4920006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:46.469 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.468+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f492c014420 con 0x7f4930060200 2023-12-08T10:09:46.670 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.669+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f493010a300 con 0x7f4930060200 2023-12-08T10:09:46.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.670+0000 7f4925ffb700 1 -- 172.21.15.170:0/541819416 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f492c010230 con 0x7f4930060200 2023-12-08T10:09:46.671 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:46.671 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.671+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f491c02ff60 msgr2=0x7f491c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.671+0000 7f4937746700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f491c02ff60 0x7f491c032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f49200096a0 tx=0x7f4920006b40).stop 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.671+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 msgr2=0x7f49301088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.671+0000 7f4937746700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49301088d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f492c013510 tx=0x7f492c00b040).stop 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.671+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 shutdown_connections 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.672+0000 7f4937746700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f491c02ff60 0x7f491c032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.672+0000 7f4937746700 1 --2- 172.21.15.170:0/541819416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4930060200 0x7f49301088d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.672+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 >> 172.21.15.170:0/541819416 conn(0x7f49300f0b70 msgr2=0x7f49300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.672+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 shutdown_connections 2023-12-08T10:09:46.673 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:46.672+0000 7f4937746700 1 -- 172.21.15.170:0/541819416 wait complete. 2023-12-08T10:09:46.674 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:47.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:46.670680+0000 mon.smithi023 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.170:0/541819416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:48.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:48.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:48.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:48 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:46.335993+0000 mgr.smithi023.tgiigg (mgr.14164) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:49.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3998342001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 msgr2=0x7fcb900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:49.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 --2- 172.21.15.170:0/3998342001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb900f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fcb84009fb0 tx=0x7fcb84009b30).stop 2023-12-08T10:09:49.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3998342001 shutdown_connections 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 --2- 172.21.15.170:0/3998342001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb900f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3998342001 >> 172.21.15.170:0/3998342001 conn(0x7fcb900f0b70 msgr2=0x7fcb900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3998342001 shutdown_connections 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3998342001 wait complete. 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.219+0000 7fcb95fe5700 1 Processor -- start 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb95fe5700 1 -- start start 2023-12-08T10:09:49.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb95fe5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb95fe5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb90106c20 con 0x7fcb900f5410 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb8f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb8f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40266/0 (socket says 172.21.15.170:40266) 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb8f7fe700 1 -- 172.21.15.170:0/3960550410 learned_addr learned my addr 172.21.15.170:0/3960550410 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb8f7fe700 1 -- 172.21.15.170:0/3960550410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb84005040 con 0x7fcb900f5410 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.220+0000 7fcb8f7fe700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb90106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcb84013510 tx=0x7fcb8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:49.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb84009320 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb84013f00 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb90106e20 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb84014d90 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb901072b0 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb84014420 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.221+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb90045ad0 con 0x7fcb900f5410 2023-12-08T10:09:49.227 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.222+0000 7fcb77fff700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcb7802ffb0 0x7fcb78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:49.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.222+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb84040fd0 con 0x7fcb900f5410 2023-12-08T10:09:49.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.222+0000 7fcb8effd700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcb7802ffb0 0x7fcb78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:49.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.222+0000 7fcb8effd700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcb7802ffb0 0x7fcb78032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fcb800096a0 tx=0x7fcb80006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:49.228 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.226+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb8406d2b0 con 0x7fcb900f5410 2023-12-08T10:09:49.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.435+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb900fae60 con 0x7fcb900f5410 2023-12-08T10:09:49.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.436+0000 7fcb77fff700 1 -- 172.21.15.170:0/3960550410 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb840095f0 con 0x7fcb900f5410 2023-12-08T10:09:49.438 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:49.438 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:49.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcb7802ffb0 msgr2=0x7fcb78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:49.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcb7802ffb0 0x7fcb78032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fcb800096a0 tx=0x7fcb80006b40).stop 2023-12-08T10:09:49.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 msgr2=0x7fcb90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:49.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb90106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcb84013510 tx=0x7fcb8400b040).stop 2023-12-08T10:09:49.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 shutdown_connections 2023-12-08T10:09:49.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcb7802ffb0 0x7fcb78032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:49.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 --2- 172.21.15.170:0/3960550410 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb900f5410 0x7fcb90106730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:49.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 >> 172.21.15.170:0/3960550410 conn(0x7fcb900f0b70 msgr2=0x7fcb900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:49.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 shutdown_connections 2023-12-08T10:09:49.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:49.438+0000 7fcb95fe5700 1 -- 172.21.15.170:0/3960550410 wait complete. 2023-12-08T10:09:49.441 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:48.336378+0000 mgr.smithi023.tgiigg (mgr.14164) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:49.436702+0000 mon.smithi023 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.170:0/3960550410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:50.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:50.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:51.998 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 -- 172.21.15.170:0/444557831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 msgr2=0x7f0ba00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:51.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 --2- 172.21.15.170:0/444557831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba00f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f0b94009fb0 tx=0x7f0b94009b30).stop 2023-12-08T10:09:51.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 -- 172.21.15.170:0/444557831 shutdown_connections 2023-12-08T10:09:51.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 --2- 172.21.15.170:0/444557831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba00f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:51.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 -- 172.21.15.170:0/444557831 >> 172.21.15.170:0/444557831 conn(0x7f0ba00f1c70 msgr2=0x7f0ba00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:51.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 -- 172.21.15.170:0/444557831 shutdown_connections 2023-12-08T10:09:51.999 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.997+0000 7f0ba5978700 1 -- 172.21.15.170:0/444557831 wait complete. 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.998+0000 7f0ba5978700 1 Processor -- start 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.998+0000 7f0ba5978700 1 -- start start 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.998+0000 7f0ba5978700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.998+0000 7f0ba5978700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba0106c20 con 0x7f0ba00f6510 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.998+0000 7f0b9effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b9effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40288/0 (socket says 172.21.15.170:40288) 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b9effd700 1 -- 172.21.15.170:0/4150998653 learned_addr learned my addr 172.21.15.170:0/4150998653 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b9effd700 1 -- 172.21.15.170:0/4150998653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94005040 con 0x7f0ba00f6510 2023-12-08T10:09:52.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b9effd700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba0106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0b94013510 tx=0x7f0b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:52.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b94009320 con 0x7f0ba00f6510 2023-12-08T10:09:52.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba0106e20 con 0x7f0ba00f6510 2023-12-08T10:09:52.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b94013f00 con 0x7f0ba00f6510 2023-12-08T10:09:52.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b94014d90 con 0x7f0ba00f6510 2023-12-08T10:09:52.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:51.999+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba01072b0 con 0x7f0ba00f6510 2023-12-08T10:09:52.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.000+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba0045ad0 con 0x7f0ba00f6510 2023-12-08T10:09:52.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.000+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b940095f0 con 0x7f0ba00f6510 2023-12-08T10:09:52.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.000+0000 7f0b877fe700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8802ff60 0x7f0b88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:52.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.000+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b94040fe0 con 0x7f0ba00f6510 2023-12-08T10:09:52.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.000+0000 7f0b9e7fc700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8802ff60 0x7f0b88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:52.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.001+0000 7f0b9e7fc700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8802ff60 0x7f0b88032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f0b900096a0 tx=0x7f0b90006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:52.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.005+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b94014420 con 0x7f0ba00f6510 2023-12-08T10:09:52.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.213+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ba00fbf60 con 0x7f0ba00f6510 2023-12-08T10:09:52.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.214+0000 7f0b877fe700 1 -- 172.21.15.170:0/4150998653 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b94010230 con 0x7f0ba00f6510 2023-12-08T10:09:52.215 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:52.216 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8802ff60 msgr2=0x7f0b88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8802ff60 0x7f0b88032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f0b900096a0 tx=0x7f0b90006b40).stop 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 msgr2=0x7f0ba0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba0106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f0b94013510 tx=0x7f0b9400b040).stop 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 shutdown_connections 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b8802ff60 0x7f0b88032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 --2- 172.21.15.170:0/4150998653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba00f6510 0x7f0ba0106730 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 >> 172.21.15.170:0/4150998653 conn(0x7f0ba00f1c70 msgr2=0x7f0ba00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:52.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 shutdown_connections 2023-12-08T10:09:52.218 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:52.216+0000 7f0ba5978700 1 -- 172.21.15.170:0/4150998653 wait complete. 2023-12-08T10:09:52.218 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:52.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:52 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:50.336666+0000 mgr.smithi023.tgiigg (mgr.14164) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:53.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:52.214719+0000 mon.smithi023 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.170:0/4150998653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:53.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:53.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:54 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:52.336989+0000 mgr.smithi023.tgiigg (mgr.14164) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 -- 172.21.15.170:0/3294598815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 msgr2=0x7f03480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 --2- 172.21.15.170:0/3294598815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f03480f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f033c009fb0 tx=0x7f033c009b30).stop 2023-12-08T10:09:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 -- 172.21.15.170:0/3294598815 shutdown_connections 2023-12-08T10:09:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 --2- 172.21.15.170:0/3294598815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f03480f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:54.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 -- 172.21.15.170:0/3294598815 >> 172.21.15.170:0/3294598815 conn(0x7f03480f1c70 msgr2=0x7f03480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 -- 172.21.15.170:0/3294598815 shutdown_connections 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.792+0000 7f03507ee700 1 -- 172.21.15.170:0/3294598815 wait complete. 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f03507ee700 1 Processor -- start 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f03507ee700 1 -- start start 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f03507ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f034806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f03507ee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348069e20 con 0x7f03480f6510 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f034e58a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f034806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f034e58a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f034806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54290/0 (socket says 172.21.15.170:54290) 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.793+0000 7f034e58a700 1 -- 172.21.15.170:0/3340188442 learned_addr learned my addr 172.21.15.170:0/3340188442 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f034e58a700 1 -- 172.21.15.170:0/3340188442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c005040 con 0x7f03480f6510 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f034e58a700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f034806b9a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f033c013510 tx=0x7f033c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f033c009320 con 0x7f03480f6510 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0348069fc0 con 0x7f03480f6510 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f033c013f00 con 0x7f03480f6510 2023-12-08T10:09:54.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f033c014d90 con 0x7f03480f6510 2023-12-08T10:09:54.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.794+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034806a450 con 0x7f03480f6510 2023-12-08T10:09:54.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.795+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f033c0095f0 con 0x7f03480f6510 2023-12-08T10:09:54.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.795+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0348059100 con 0x7f03480f6510 2023-12-08T10:09:54.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.795+0000 7f033affd700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f033402ff60 0x7f0334032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:54.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.795+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f033c040fc0 con 0x7f03480f6510 2023-12-08T10:09:54.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.795+0000 7f034dd89700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f033402ff60 0x7f0334032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:54.798 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.795+0000 7f034dd89700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f033402ff60 0x7f0334032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f03440096a0 tx=0x7f0344006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:54.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:54.799+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f033c014420 con 0x7f03480f6510 2023-12-08T10:09:55.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.004+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0348045ad0 con 0x7f03480f6510 2023-12-08T10:09:55.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.005+0000 7f033affd700 1 -- 172.21.15.170:0/3340188442 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f033c010230 con 0x7f03480f6510 2023-12-08T10:09:55.007 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:55.007 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f033402ff60 msgr2=0x7f0334032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f033402ff60 0x7f0334032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f03440096a0 tx=0x7f0344006b40).stop 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 msgr2=0x7f034806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f034806b9a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f033c013510 tx=0x7f033c00b040).stop 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 shutdown_connections 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f033402ff60 0x7f0334032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:55.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 --2- 172.21.15.170:0/3340188442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03480f6510 0x7f034806b9a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:55.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 >> 172.21.15.170:0/3340188442 conn(0x7f03480f1c70 msgr2=0x7f03480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:55.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 shutdown_connections 2023-12-08T10:09:55.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:55.007+0000 7f03507ee700 1 -- 172.21.15.170:0/3340188442 wait complete. 2023-12-08T10:09:55.009 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:55.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:55.005866+0000 mon.smithi023 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.170:0/3340188442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:56.399 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:56.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:09:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:56 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:54.337284+0000 mgr.smithi023.tgiigg (mgr.14164) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 -- 172.21.15.170:0/3559739551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 msgr2=0x7f4d4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 --2- 172.21.15.170:0/3559739551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c0f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f4d34009fb0 tx=0x7f4d34009b30).stop 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 -- 172.21.15.170:0/3559739551 shutdown_connections 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 --2- 172.21.15.170:0/3559739551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c0f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 -- 172.21.15.170:0/3559739551 >> 172.21.15.170:0/3559739551 conn(0x7f4d4c0f0b70 msgr2=0x7f4d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 -- 172.21.15.170:0/3559739551 shutdown_connections 2023-12-08T10:09:57.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.574+0000 7f4d510aa700 1 -- 172.21.15.170:0/3559739551 wait complete. 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d510aa700 1 Processor -- start 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d510aa700 1 -- start start 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d510aa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d510aa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c060220 con 0x7f4d4c0f5410 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d4ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d4ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54312/0 (socket says 172.21.15.170:54312) 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.575+0000 7f4d4ad9d700 1 -- 172.21.15.170:0/684865476 learned_addr learned my addr 172.21.15.170:0/684865476 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d4ad9d700 1 -- 172.21.15.170:0/684865476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d34005040 con 0x7f4d4c0f5410 2023-12-08T10:09:57.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d4ad9d700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c109450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4d34013510 tx=0x7f4d3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d34009320 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d34013f00 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c060420 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d34014d90 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.576+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c109b00 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.577+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d340095f0 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.577+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c059100 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.577+0000 7f4d437fe700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d3802ff60 0x7f4d38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.577+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d340410c0 con 0x7f4d4c0f5410 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.577+0000 7f4d4a59c700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d3802ff60 0x7f4d38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:09:57.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.578+0000 7f4d4a59c700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d3802ff60 0x7f4d38032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f4d3c0096a0 tx=0x7f4d3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:09:57.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.582+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d34014420 con 0x7f4d4c0f5410 2023-12-08T10:09:57.788 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.787+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d4c045ad0 con 0x7f4d4c0f5410 2023-12-08T10:09:57.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.788+0000 7f4d437fe700 1 -- 172.21.15.170:0/684865476 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d3401d7f0 con 0x7f4d4c0f5410 2023-12-08T10:09:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:09:57.789 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:09:57.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.789+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d3802ff60 msgr2=0x7f4d38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:57.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.789+0000 7f4d510aa700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d3802ff60 0x7f4d38032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f4d3c0096a0 tx=0x7f4d3c006b40).stop 2023-12-08T10:09:57.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 msgr2=0x7f4d4c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:09:57.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c109450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4d34013510 tx=0x7f4d3400b040).stop 2023-12-08T10:09:57.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 shutdown_connections 2023-12-08T10:09:57.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d3802ff60 0x7f4d38032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:57.792 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 --2- 172.21.15.170:0/684865476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d4c0f5410 0x7f4d4c109450 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:09:57.792 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 >> 172.21.15.170:0/684865476 conn(0x7f4d4c0f0b70 msgr2=0x7f4d4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:09:57.792 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 shutdown_connections 2023-12-08T10:09:57.792 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:09:57.790+0000 7f4d510aa700 1 -- 172.21.15.170:0/684865476 wait complete. 2023-12-08T10:09:57.792 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:09:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:56.337574+0000 mgr.smithi023.tgiigg (mgr.14164) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:09:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:09:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:09:57.788558+0000 mon.smithi023 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.170:0/684865476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:09:59.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:09:59.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:00.285 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/2174345938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 msgr2=0x7fcd580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 --2- 172.21.15.170:0/2174345938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd580f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fcd40009fb0 tx=0x7fcd40009b30).stop 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/2174345938 shutdown_connections 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 --2- 172.21.15.170:0/2174345938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd580f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/2174345938 >> 172.21.15.170:0/2174345938 conn(0x7fcd580f0b70 msgr2=0x7fcd580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/2174345938 shutdown_connections 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.282+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/2174345938 wait complete. 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd5d3b8700 1 Processor -- start 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd5d3b8700 1 -- start start 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd5d3b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd58105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd5d3b8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58105af0 con 0x7fcd580f5410 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd56ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd58105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd56ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd58105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54330/0 (socket says 172.21.15.170:54330) 2023-12-08T10:10:00.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.283+0000 7fcd56ffd700 1 -- 172.21.15.170:0/3676216937 learned_addr learned my addr 172.21.15.170:0/3676216937 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd56ffd700 1 -- 172.21.15.170:0/3676216937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd40005040 con 0x7fcd580f5410 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd56ffd700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd58105600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fcd40013510 tx=0x7fcd4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd40009320 con 0x7fcd580f5410 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd58105cf0 con 0x7fcd580f5410 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd40013f00 con 0x7fcd580f5410 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd40014d90 con 0x7fcd580f5410 2023-12-08T10:10:00.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.284+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd58106180 con 0x7fcd580f5410 2023-12-08T10:10:00.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.285+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd40014420 con 0x7fcd580f5410 2023-12-08T10:10:00.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.285+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd58045ad0 con 0x7fcd580f5410 2023-12-08T10:10:00.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.285+0000 7fcd4f7fe700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcd4402ff60 0x7fcd44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:00.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.285+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd40041000 con 0x7fcd580f5410 2023-12-08T10:10:00.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.285+0000 7fcd567fc700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcd4402ff60 0x7fcd44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:00.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.286+0000 7fcd567fc700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcd4402ff60 0x7fcd44032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fcd480096a0 tx=0x7fcd48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:00.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.290+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd400095f0 con 0x7fcd580f5410 2023-12-08T10:10:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:00 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:09:58.337908+0000 mgr.smithi023.tgiigg (mgr.14164) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:00.499 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.498+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd580fae60 con 0x7fcd580f5410 2023-12-08T10:10:00.500 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.498+0000 7fcd4f7fe700 1 -- 172.21.15.170:0/3676216937 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd4001db70 con 0x7fcd580f5410 2023-12-08T10:10:00.501 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:00.501 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:00.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcd4402ff60 msgr2=0x7fcd44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:00.501 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcd4402ff60 0x7fcd44032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fcd480096a0 tx=0x7fcd48006b40).stop 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 msgr2=0x7fcd58105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd58105600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fcd40013510 tx=0x7fcd4000b040).stop 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 shutdown_connections 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fcd4402ff60 0x7fcd44032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 --2- 172.21.15.170:0/3676216937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd580f5410 0x7fcd58105600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 >> 172.21.15.170:0/3676216937 conn(0x7fcd580f0b70 msgr2=0x7fcd580f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 shutdown_connections 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:00.500+0000 7fcd5d3b8700 1 -- 172.21.15.170:0/3676216937 wait complete. 2023-12-08T10:10:00.502 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:01.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:00.499081+0000 mon.smithi023 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.170:0/3676216937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:01.892 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:01.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:00.338300+0000 mgr.smithi023.tgiigg (mgr.14164) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 -- 172.21.15.170:0/4223767380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 msgr2=0x7fd8f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 --2- 172.21.15.170:0/4223767380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f40f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fd8e8009fb0 tx=0x7fd8e8009b30).stop 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 -- 172.21.15.170:0/4223767380 shutdown_connections 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 --2- 172.21.15.170:0/4223767380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f40f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 -- 172.21.15.170:0/4223767380 >> 172.21.15.170:0/4223767380 conn(0x7fd8f40f1c70 msgr2=0x7fd8f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 -- 172.21.15.170:0/4223767380 shutdown_connections 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.029+0000 7fd8f8b24700 1 -- 172.21.15.170:0/4223767380 wait complete. 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f8b24700 1 Processor -- start 2023-12-08T10:10:03.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f8b24700 1 -- start start 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f8b24700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f8b24700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4106c20 con 0x7fd8f40f6510 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40492/0 (socket says 172.21.15.170:40492) 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f259c700 1 -- 172.21.15.170:0/3139354257 learned_addr learned my addr 172.21.15.170:0/3139354257 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f259c700 1 -- 172.21.15.170:0/3139354257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8005040 con 0x7fd8f40f6510 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.030+0000 7fd8f259c700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f4106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fd8e8013510 tx=0x7fd8e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.031+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e8009320 con 0x7fd8f40f6510 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.031+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8e8013f00 con 0x7fd8f40f6510 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.031+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f4106e20 con 0x7fd8f40f6510 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.031+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e8014d90 con 0x7fd8f40f6510 2023-12-08T10:10:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.031+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f41072b0 con 0x7fd8f40f6510 2023-12-08T10:10:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.031+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8e80095f0 con 0x7fd8f40f6510 2023-12-08T10:10:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.032+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f4045ad0 con 0x7fd8f40f6510 2023-12-08T10:10:03.034 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.032+0000 7fd8e2ffd700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd8dc02ff60 0x7fd8dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:03.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.032+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8e8041020 con 0x7fd8f40f6510 2023-12-08T10:10:03.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.032+0000 7fd8f1d9b700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd8dc02ff60 0x7fd8dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:03.035 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.032+0000 7fd8f1d9b700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd8dc02ff60 0x7fd8dc032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd8e40096a0 tx=0x7fd8e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:03.038 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.037+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8e8014420 con 0x7fd8f40f6510 2023-12-08T10:10:03.245 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.243+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8f40fbf60 con 0x7fd8f40f6510 2023-12-08T10:10:03.245 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.244+0000 7fd8e2ffd700 1 -- 172.21.15.170:0/3139354257 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd8e801d7f0 con 0x7fd8f40f6510 2023-12-08T10:10:03.246 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:03.246 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:03.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd8dc02ff60 msgr2=0x7fd8dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:03.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd8dc02ff60 0x7fd8dc032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fd8e40096a0 tx=0x7fd8e4006b40).stop 2023-12-08T10:10:03.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 msgr2=0x7fd8f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:03.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f4106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fd8e8013510 tx=0x7fd8e800b040).stop 2023-12-08T10:10:03.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 shutdown_connections 2023-12-08T10:10:03.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd8dc02ff60 0x7fd8dc032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:03.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 --2- 172.21.15.170:0/3139354257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8f40f6510 0x7fd8f4106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:03.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 >> 172.21.15.170:0/3139354257 conn(0x7fd8f40f1c70 msgr2=0x7fd8f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:03.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 shutdown_connections 2023-12-08T10:10:03.248 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:03.246+0000 7fd8f8b24700 1 -- 172.21.15.170:0/3139354257 wait complete. 2023-12-08T10:10:03.248 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:02.338619+0000 mgr.smithi023.tgiigg (mgr.14164) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:03.244871+0000 mon.smithi023 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.170:0/3139354257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:04.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:04.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 -- 172.21.15.170:0/3143426436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 msgr2=0x7f7cf80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 --2- 172.21.15.170:0/3143426436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf80f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f7cec009fb0 tx=0x7f7cec009b30).stop 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 -- 172.21.15.170:0/3143426436 shutdown_connections 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 --2- 172.21.15.170:0/3143426436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf80f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 -- 172.21.15.170:0/3143426436 >> 172.21.15.170:0/3143426436 conn(0x7f7cf80f09a0 msgr2=0x7f7cf80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 -- 172.21.15.170:0/3143426436 shutdown_connections 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.790+0000 7f7cfdfac700 1 -- 172.21.15.170:0/3143426436 wait complete. 2023-12-08T10:10:05.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.791+0000 7f7cfdfac700 1 Processor -- start 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.791+0000 7f7cfdfac700 1 -- start start 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.791+0000 7f7cfdfac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.791+0000 7f7cfdfac700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8060240 con 0x7f7cf80f51a0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfcfaa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfcfaa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40514/0 (socket says 172.21.15.170:40514) 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfcfaa700 1 -- 172.21.15.170:0/937447613 learned_addr learned my addr 172.21.15.170:0/937447613 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfcfaa700 1 -- 172.21.15.170:0/937447613 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec005040 con 0x7f7cf80f51a0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfcfaa700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f7cec013510 tx=0x7f7cec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cec009320 con 0x7f7cf80f51a0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf8060440 con 0x7f7cf80f51a0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cec013f00 con 0x7f7cf80f51a0 2023-12-08T10:10:05.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cec014dc0 con 0x7f7cf80f51a0 2023-12-08T10:10:05.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.792+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf8109860 con 0x7f7cf80f51a0 2023-12-08T10:10:05.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.793+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cec01d3d0 con 0x7f7cf80f51a0 2023-12-08T10:10:05.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.793+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf8045ad0 con 0x7f7cf80f51a0 2023-12-08T10:10:05.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.793+0000 7f7cf57fa700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:05.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.793+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cec041410 con 0x7f7cf80f51a0 2023-12-08T10:10:05.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.794+0000 7f7cf7fff700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:05.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.794+0000 7f7cf7fff700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ce802ff60 0x7f7ce8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7ce40096a0 tx=0x7f7ce4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:05.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:05.798+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cec01cad0 con 0x7f7cf80f51a0 2023-12-08T10:10:06.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.004+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7cf810a080 con 0x7f7cf80f51a0 2023-12-08T10:10:06.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.005+0000 7f7cf57fa700 1 -- 172.21.15.170:0/937447613 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7cec01cad0 con 0x7f7cf80f51a0 2023-12-08T10:10:06.006 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:06.006 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ce802ff60 msgr2=0x7f7ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ce802ff60 0x7f7ce8032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7ce40096a0 tx=0x7f7ce4006b40).stop 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 msgr2=0x7f7cf81091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f7cec013510 tx=0x7f7cec00b040).stop 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 shutdown_connections 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ce802ff60 0x7f7ce8032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 --2- 172.21.15.170:0/937447613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf80f51a0 0x7f7cf81091b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 >> 172.21.15.170:0/937447613 conn(0x7f7cf80f09a0 msgr2=0x7f7cf80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 shutdown_connections 2023-12-08T10:10:06.009 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:06.007+0000 7f7cfdfac700 1 -- 172.21.15.170:0/937447613 wait complete. 2023-12-08T10:10:06.009 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:04.338928+0000 mgr.smithi023.tgiigg (mgr.14164) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:06.005809+0000 mon.smithi023 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.170:0/937447613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:07.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:07.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:08 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:06.339271+0000 mgr.smithi023.tgiigg (mgr.14164) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 -- 172.21.15.170:0/3449729430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 msgr2=0x7f47a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 --2- 172.21.15.170:0/3449729430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a80f90f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f4790009fb0 tx=0x7f4790009b30).stop 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 -- 172.21.15.170:0/3449729430 shutdown_connections 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 --2- 172.21.15.170:0/3449729430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a80f90f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 -- 172.21.15.170:0/3449729430 >> 172.21.15.170:0/3449729430 conn(0x7f47a80f0b70 msgr2=0x7f47a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 -- 172.21.15.170:0/3449729430 shutdown_connections 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 -- 172.21.15.170:0/3449729430 wait complete. 2023-12-08T10:10:08.625 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.621+0000 7f47ae016700 1 Processor -- start 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47ae016700 1 -- start start 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47ae016700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a81088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47ae016700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a8108de0 con 0x7f47a8060280 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47a77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a81088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47a77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a81088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40544/0 (socket says 172.21.15.170:40544) 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47a77fe700 1 -- 172.21.15.170:0/2682311537 learned_addr learned my addr 172.21.15.170:0/2682311537 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47a77fe700 1 -- 172.21.15.170:0/2682311537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790005040 con 0x7f47a8060280 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.622+0000 7f47a77fe700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a81088f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f4790013510 tx=0x7f479000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4790009320 con 0x7f47a8060280 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4790013f00 con 0x7f47a8060280 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a8108fe0 con 0x7f47a8060280 2023-12-08T10:10:08.626 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4790014dc0 con 0x7f47a8060280 2023-12-08T10:10:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a8109470 con 0x7f47a8060280 2023-12-08T10:10:08.627 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4790014420 con 0x7f47a8060280 2023-12-08T10:10:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8045ad0 con 0x7f47a8060280 2023-12-08T10:10:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.623+0000 7f479ffff700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f479402ff60 0x7f4794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.624+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4790041050 con 0x7f47a8060280 2023-12-08T10:10:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.624+0000 7f47a6ffd700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f479402ff60 0x7f4794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:08.628 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.624+0000 7f47a6ffd700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f479402ff60 0x7f4794032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f47980096a0 tx=0x7f4798006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:08.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.628+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47900095f0 con 0x7f47a8060280 2023-12-08T10:10:08.835 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.833+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a810a320 con 0x7f47a8060280 2023-12-08T10:10:08.836 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.834+0000 7f479ffff700 1 -- 172.21.15.170:0/2682311537 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f479001cb10 con 0x7f47a8060280 2023-12-08T10:10:08.836 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:08.836 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:08.838 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f479402ff60 msgr2=0x7f4794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:08.838 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f479402ff60 0x7f4794032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f47980096a0 tx=0x7f4798006b40).stop 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 msgr2=0x7f47a81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a81088f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f4790013510 tx=0x7f479000b040).stop 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 shutdown_connections 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f479402ff60 0x7f4794032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 --2- 172.21.15.170:0/2682311537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47a8060280 0x7f47a81088f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 >> 172.21.15.170:0/2682311537 conn(0x7f47a80f0b70 msgr2=0x7f47a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 shutdown_connections 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:08.835+0000 7f47ae016700 1 -- 172.21.15.170:0/2682311537 wait complete. 2023-12-08T10:10:08.839 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:09.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:08.835441+0000 mon.smithi023 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.170:0/2682311537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:10.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:10.222 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:08.339651+0000 mgr.smithi023.tgiigg (mgr.14164) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 -- 172.21.15.170:0/1931756747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 msgr2=0x7fe5080f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 --2- 172.21.15.170:0/1931756747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe5080f66e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe4fc009fb0 tx=0x7fe4fc009b30).stop 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 -- 172.21.15.170:0/1931756747 shutdown_connections 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 --2- 172.21.15.170:0/1931756747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe5080f66e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 -- 172.21.15.170:0/1931756747 >> 172.21.15.170:0/1931756747 conn(0x7fe5080f1ac0 msgr2=0x7fe5080f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 -- 172.21.15.170:0/1931756747 shutdown_connections 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 -- 172.21.15.170:0/1931756747 wait complete. 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 Processor -- start 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.338+0000 7fe50dbfd700 1 -- start start 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50dbfd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe508106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:11.341 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50dbfd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe508106a10 con 0x7fe5080f62c0 2023-12-08T10:10:11.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50cbfb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe508106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:11.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50cbfb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe508106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:40562/0 (socket says 172.21.15.170:40562) 2023-12-08T10:10:11.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50cbfb700 1 -- 172.21.15.170:0/972910731 learned_addr learned my addr 172.21.15.170:0/972910731 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:11.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50cbfb700 1 -- 172.21.15.170:0/972910731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc005040 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.339+0000 7fe50cbfb700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe508106520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fe4fc013510 tx=0x7fe4fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4fc009320 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe508106c10 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5081070a0 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4fc013f00 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4fc014d90 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4fc014420 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.340+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e8004fa0 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.341+0000 7fe5057fa700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe4f802ff60 0x7fe4f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.341+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4fc0412e0 con 0x7fe5080f62c0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.341+0000 7fe507fff700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe4f802ff60 0x7fe4f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:11.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.341+0000 7fe507fff700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe4f802ff60 0x7fe4f8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe4f40096a0 tx=0x7fe4f4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:11.347 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.345+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4fc012b20 con 0x7fe5080f62c0 2023-12-08T10:10:11.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:10.340015+0000 mgr.smithi023.tgiigg (mgr.14164) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:11.557 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.555+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4e8005670 con 0x7fe5080f62c0 2023-12-08T10:10:11.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.556+0000 7fe5057fa700 1 -- 172.21.15.170:0/972910731 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe4fc012b20 con 0x7fe5080f62c0 2023-12-08T10:10:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:11.558 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:11.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe4f802ff60 msgr2=0x7fe4f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe4f802ff60 0x7fe4f8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fe4f40096a0 tx=0x7fe4f4006b40).stop 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 msgr2=0x7fe508106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe508106520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fe4fc013510 tx=0x7fe4fc00b040).stop 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 shutdown_connections 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe4f802ff60 0x7fe4f8032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 --2- 172.21.15.170:0/972910731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5080f62c0 0x7fe508106520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 >> 172.21.15.170:0/972910731 conn(0x7fe5080f1ac0 msgr2=0x7fe5080f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 shutdown_connections 2023-12-08T10:10:11.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:11.557+0000 7fe50dbfd700 1 -- 172.21.15.170:0/972910731 wait complete. 2023-12-08T10:10:11.561 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:12.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:11.557264+0000 mon.smithi023 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.170:0/972910731' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:12.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:12.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.063+0000 7f7c029b0700 1 -- 172.21.15.170:0/11520120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 msgr2=0x7f7bfc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.063+0000 7f7c029b0700 1 --2- 172.21.15.170:0/11520120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc0f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7bec009fb0 tx=0x7f7bec009b30).stop 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.063+0000 7f7c029b0700 1 -- 172.21.15.170:0/11520120 shutdown_connections 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.063+0000 7f7c029b0700 1 --2- 172.21.15.170:0/11520120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc0f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.063+0000 7f7c029b0700 1 -- 172.21.15.170:0/11520120 >> 172.21.15.170:0/11520120 conn(0x7f7bfc0f0b70 msgr2=0x7f7bfc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.064+0000 7f7c029b0700 1 -- 172.21.15.170:0/11520120 shutdown_connections 2023-12-08T10:10:14.066 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.064+0000 7f7c029b0700 1 -- 172.21.15.170:0/11520120 wait complete. 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.064+0000 7f7c029b0700 1 Processor -- start 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.064+0000 7f7c029b0700 1 -- start start 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7c029b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7c029b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc105b20 con 0x7f7bfc0f5410 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7bfbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7bfbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34898/0 (socket says 172.21.15.170:34898) 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7bfbfff700 1 -- 172.21.15.170:0/919707601 learned_addr learned my addr 172.21.15.170:0/919707601 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7bfbfff700 1 -- 172.21.15.170:0/919707601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec005040 con 0x7f7bfc0f5410 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.065+0000 7f7bfbfff700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc105630 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7bec013510 tx=0x7f7bec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:14.068 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.066+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bec009320 con 0x7f7bfc0f5410 2023-12-08T10:10:14.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.066+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bec013f00 con 0x7f7bfc0f5410 2023-12-08T10:10:14.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.066+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bfc105d20 con 0x7f7bfc0f5410 2023-12-08T10:10:14.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.066+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bec014d90 con 0x7f7bfc0f5410 2023-12-08T10:10:14.069 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.066+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bfc1061b0 con 0x7f7bfc0f5410 2023-12-08T10:10:14.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.067+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bec014420 con 0x7f7bfc0f5410 2023-12-08T10:10:14.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.067+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bfc045ad0 con 0x7f7bfc0f5410 2023-12-08T10:10:14.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.067+0000 7f7bf8ff9700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7be402ff60 0x7f7be4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:14.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.067+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bec041000 con 0x7f7bfc0f5410 2023-12-08T10:10:14.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.067+0000 7f7bfb7fe700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7be402ff60 0x7f7be4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:14.070 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.067+0000 7f7bfb7fe700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7be402ff60 0x7f7be4032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7bf00096a0 tx=0x7f7bf0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:14.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.072+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bec0095f0 con 0x7f7bfc0f5410 2023-12-08T10:10:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:12.340361+0000 mgr.smithi023.tgiigg (mgr.14164) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:14.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.278+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bfc0fae60 con 0x7f7bfc0f5410 2023-12-08T10:10:14.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.278+0000 7f7bf8ff9700 1 -- 172.21.15.170:0/919707601 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bec01db70 con 0x7f7bfc0f5410 2023-12-08T10:10:14.281 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:14.281 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7be402ff60 msgr2=0x7f7be4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7be402ff60 0x7f7be4032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7bf00096a0 tx=0x7f7bf0006b40).stop 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 msgr2=0x7f7bfc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc105630 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7bec013510 tx=0x7f7bec00b040).stop 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 shutdown_connections 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7be402ff60 0x7f7be4032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 --2- 172.21.15.170:0/919707601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bfc0f5410 0x7f7bfc105630 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:14.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 >> 172.21.15.170:0/919707601 conn(0x7f7bfc0f0b70 msgr2=0x7f7bfc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:14.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 shutdown_connections 2023-12-08T10:10:14.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:14.280+0000 7f7c029b0700 1 -- 172.21.15.170:0/919707601 wait complete. 2023-12-08T10:10:14.283 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:15.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:14.279967+0000 mon.smithi023 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.170:0/919707601' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:15.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:15.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:14.340648+0000 mgr.smithi023.tgiigg (mgr.14164) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:16.807 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 -- 172.21.15.170:0/4057530134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 msgr2=0x7f613c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 --2- 172.21.15.170:0/4057530134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c0f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f6130009fb0 tx=0x7f6130009b30).stop 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 -- 172.21.15.170:0/4057530134 shutdown_connections 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 --2- 172.21.15.170:0/4057530134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c0f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 -- 172.21.15.170:0/4057530134 >> 172.21.15.170:0/4057530134 conn(0x7f613c0f0b70 msgr2=0x7f613c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 -- 172.21.15.170:0/4057530134 shutdown_connections 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.804+0000 7f6143336700 1 -- 172.21.15.170:0/4057530134 wait complete. 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f6143336700 1 Processor -- start 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f6143336700 1 -- start start 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f6143336700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c1071e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f6143336700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613c060280 con 0x7f613c0f5410 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f61410d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c1071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f61410d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c1071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34920/0 (socket says 172.21.15.170:34920) 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f61410d2700 1 -- 172.21.15.170:0/2893453294 learned_addr learned my addr 172.21.15.170:0/2893453294 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:16.808 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f61410d2700 1 -- 172.21.15.170:0/2893453294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6130005040 con 0x7f613c0f5410 2023-12-08T10:10:16.809 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.805+0000 7f61410d2700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c1071e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6130013510 tx=0x7f613000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:16.810 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.806+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6130009320 con 0x7f613c0f5410 2023-12-08T10:10:16.810 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.806+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6130013f00 con 0x7f613c0f5410 2023-12-08T10:10:16.810 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.806+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c060480 con 0x7f613c0f5410 2023-12-08T10:10:16.810 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.806+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6130014d90 con 0x7f613c0f5410 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.806+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613c107910 con 0x7f613c0f5410 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.807+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61300095f0 con 0x7f613c0f5410 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.807+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613c045ad0 con 0x7f613c0f5410 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.807+0000 7f612dffb700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f612802ffb0 0x7f6128032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.807+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6130041080 con 0x7f613c0f5410 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.807+0000 7f61408d1700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f612802ffb0 0x7f6128032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:16.811 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.807+0000 7f61408d1700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f612802ffb0 0x7f6128032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f61380096a0 tx=0x7f6138006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:16.813 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:16.811+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f613006d2b0 con 0x7f613c0f5410 2023-12-08T10:10:17.021 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.019+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f613c0fae60 con 0x7f613c0f5410 2023-12-08T10:10:17.022 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.020+0000 7f612dffb700 1 -- 172.21.15.170:0/2893453294 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6130014420 con 0x7f613c0f5410 2023-12-08T10:10:17.022 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:17.022 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f612802ffb0 msgr2=0x7f6128032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f612802ffb0 0x7f6128032470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f61380096a0 tx=0x7f6138006b40).stop 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 msgr2=0x7f613c1071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c1071e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f6130013510 tx=0x7f613000b040).stop 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 shutdown_connections 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f612802ffb0 0x7f6128032470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 --2- 172.21.15.170:0/2893453294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f613c0f5410 0x7f613c1071e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 >> 172.21.15.170:0/2893453294 conn(0x7f613c0f0b70 msgr2=0x7f613c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 shutdown_connections 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:17.021+0000 7f6143336700 1 -- 172.21.15.170:0/2893453294 wait complete. 2023-12-08T10:10:17.024 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:16.340948+0000 mgr.smithi023.tgiigg (mgr.14164) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:17.021362+0000 mon.smithi023 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.170:0/2893453294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:18.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:18.425 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:19.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 -- 172.21.15.170:0/318407293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 msgr2=0x7fa9e40f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:19.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 --2- 172.21.15.170:0/318407293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e40f8fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa9d4009fb0 tx=0x7fa9d4009b30).stop 2023-12-08T10:10:19.544 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 -- 172.21.15.170:0/318407293 shutdown_connections 2023-12-08T10:10:19.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 --2- 172.21.15.170:0/318407293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e40f8fc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:19.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 -- 172.21.15.170:0/318407293 >> 172.21.15.170:0/318407293 conn(0x7fa9e40f09a0 msgr2=0x7fa9e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:19.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 -- 172.21.15.170:0/318407293 shutdown_connections 2023-12-08T10:10:19.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 -- 172.21.15.170:0/318407293 wait complete. 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 Processor -- start 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.542+0000 7fa9e3fff700 1 -- start start 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e4106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e3fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4106a10 con 0x7fa9e4060140 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e4106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e4106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34934/0 (socket says 172.21.15.170:34934) 2023-12-08T10:10:19.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e2ffd700 1 -- 172.21.15.170:0/1370177971 learned_addr learned my addr 172.21.15.170:0/1370177971 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e2ffd700 1 -- 172.21.15.170:0/1370177971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4005040 con 0x7fa9e4060140 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.543+0000 7fa9e2ffd700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e4106520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa9d4013510 tx=0x7fa9d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9d4009320 con 0x7fa9e4060140 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e4106c10 con 0x7fa9e4060140 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9d4013f00 con 0x7fa9e4060140 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9d4014d90 con 0x7fa9e4060140 2023-12-08T10:10:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e41070a0 con 0x7fa9e4060140 2023-12-08T10:10:19.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9d40095f0 con 0x7fa9e4060140 2023-12-08T10:10:19.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.544+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e4045ad0 con 0x7fa9e4060140 2023-12-08T10:10:19.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.545+0000 7fa9cb7fe700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa9cc02ff60 0x7fa9cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:19.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.545+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9d4041000 con 0x7fa9e4060140 2023-12-08T10:10:19.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.545+0000 7fa9e27fc700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa9cc02ff60 0x7fa9cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:19.548 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.545+0000 7fa9e27fc700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa9cc02ff60 0x7fa9cc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa9d80096a0 tx=0x7fa9d8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:19.551 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.549+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9d4014420 con 0x7fa9e4060140 2023-12-08T10:10:19.758 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.756+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9e4107f50 con 0x7fa9e4060140 2023-12-08T10:10:19.759 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.757+0000 7fa9cb7fe700 1 -- 172.21.15.170:0/1370177971 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa9d4010230 con 0x7fa9e4060140 2023-12-08T10:10:19.759 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:19.759 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa9cc02ff60 msgr2=0x7fa9cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa9cc02ff60 0x7fa9cc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa9d80096a0 tx=0x7fa9d8006b40).stop 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 msgr2=0x7fa9e4106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e4106520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa9d4013510 tx=0x7fa9d400b040).stop 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 shutdown_connections 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa9cc02ff60 0x7fa9cc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 --2- 172.21.15.170:0/1370177971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9e4060140 0x7fa9e4106520 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 >> 172.21.15.170:0/1370177971 conn(0x7fa9e40f09a0 msgr2=0x7fa9e40f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 shutdown_connections 2023-12-08T10:10:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:19.758+0000 7fa9e3fff700 1 -- 172.21.15.170:0/1370177971 wait complete. 2023-12-08T10:10:19.762 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:20.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:18.341322+0000 mgr.smithi023.tgiigg (mgr.14164) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:20.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:19.758371+0000 mon.smithi023 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.170:0/1370177971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:21.140 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:21.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:20.341657+0000 mgr.smithi023.tgiigg (mgr.14164) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:22.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1147506744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 msgr2=0x7fd9e00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:22.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 --2- 172.21.15.170:0/1147506744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e00f8f80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fd9d0009fb0 tx=0x7fd9d0009b30).stop 2023-12-08T10:10:22.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1147506744 shutdown_connections 2023-12-08T10:10:22.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 --2- 172.21.15.170:0/1147506744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e00f8f80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1147506744 >> 172.21.15.170:0/1147506744 conn(0x7fd9e00f09a0 msgr2=0x7fd9e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1147506744 shutdown_connections 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.275+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1147506744 wait complete. 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.276+0000 7fd9e4d7b700 1 Processor -- start 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.276+0000 7fd9e4d7b700 1 -- start start 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.277+0000 7fd9e4d7b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.277+0000 7fd9e4d7b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e01069f0 con 0x7fd9e0060180 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.277+0000 7fd9df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.277+0000 7fd9df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54292/0 (socket says 172.21.15.170:54292) 2023-12-08T10:10:22.280 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.277+0000 7fd9df7fe700 1 -- 172.21.15.170:0/1046815777 learned_addr learned my addr 172.21.15.170:0/1046815777 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.277+0000 7fd9df7fe700 1 -- 172.21.15.170:0/1046815777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0005040 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.278+0000 7fd9df7fe700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e0106500 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fd9d0013510 tx=0x7fd9d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.278+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d0009320 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.278+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9d0013f00 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.278+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d0014d90 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.278+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e0106bf0 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.278+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e0107080 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.279+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9d00095f0 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.279+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e0045ad0 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.279+0000 7fd9c7fff700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd9c802ff60 0x7fd9c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.279+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9d0041030 con 0x7fd9e0060180 2023-12-08T10:10:22.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.279+0000 7fd9deffd700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd9c802ff60 0x7fd9c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:22.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.280+0000 7fd9deffd700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd9c802ff60 0x7fd9c8032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd9d40096a0 tx=0x7fd9d4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:22.286 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.284+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9d0014420 con 0x7fd9e0060180 2023-12-08T10:10:22.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.492+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9e0107f30 con 0x7fd9e0060180 2023-12-08T10:10:22.495 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.493+0000 7fd9c7fff700 1 -- 172.21.15.170:0/1046815777 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9d0010230 con 0x7fd9e0060180 2023-12-08T10:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:22.495 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd9c802ff60 msgr2=0x7fd9c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd9c802ff60 0x7fd9c8032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fd9d40096a0 tx=0x7fd9d4006b40).stop 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 msgr2=0x7fd9e0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e0106500 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fd9d0013510 tx=0x7fd9d000b040).stop 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 shutdown_connections 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd9c802ff60 0x7fd9c8032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 --2- 172.21.15.170:0/1046815777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9e0060180 0x7fd9e0106500 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.494+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 >> 172.21.15.170:0/1046815777 conn(0x7fd9e00f09a0 msgr2=0x7fd9e00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.495+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 shutdown_connections 2023-12-08T10:10:22.497 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:22.495+0000 7fd9e4d7b700 1 -- 172.21.15.170:0/1046815777 wait complete. 2023-12-08T10:10:22.498 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:22.452387+0000 mon.smithi023 (mon.0) 199 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:10:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:22.454823+0000 mon.smithi023 (mon.0) 200 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:10:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:22.494357+0000 mon.smithi023 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.170:0/1046815777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:23.886 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:23.887 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:22.342014+0000 mgr.smithi023.tgiigg (mgr.14164) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.050+0000 7f0676f1f700 1 -- 172.21.15.170:0/3538355216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 msgr2=0x7f06700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.050+0000 7f0676f1f700 1 --2- 172.21.15.170:0/3538355216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f06700f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f0664009fb0 tx=0x7f0664009b30).stop 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 -- 172.21.15.170:0/3538355216 shutdown_connections 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 --2- 172.21.15.170:0/3538355216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f06700f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 -- 172.21.15.170:0/3538355216 >> 172.21.15.170:0/3538355216 conn(0x7f06700f1c70 msgr2=0x7f06700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 -- 172.21.15.170:0/3538355216 shutdown_connections 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 -- 172.21.15.170:0/3538355216 wait complete. 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 Processor -- start 2023-12-08T10:10:25.054 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.051+0000 7f0676f1f700 1 -- start start 2023-12-08T10:10:25.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0676f1f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f067006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:25.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0676f1f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0670069e20 con 0x7f06700f6510 2023-12-08T10:10:25.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0674cbb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f067006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:25.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0674cbb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f067006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54308/0 (socket says 172.21.15.170:54308) 2023-12-08T10:10:25.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0674cbb700 1 -- 172.21.15.170:0/8021075 learned_addr learned my addr 172.21.15.170:0/8021075 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:25.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0674cbb700 1 -- 172.21.15.170:0/8021075 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664005040 con 0x7f06700f6510 2023-12-08T10:10:25.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.052+0000 7f0674cbb700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f067006b9a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f0664013510 tx=0x7f066400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0664009320 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0670069fc0 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0664013f00 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0664014d90 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067006a450 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0664014420 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.053+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0670059100 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.054+0000 7f066d7fa700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f066002ff60 0x7f0660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.054+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0664041110 con 0x7f06700f6510 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.054+0000 7f066ffff700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f066002ff60 0x7f0660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:25.057 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.054+0000 7f066ffff700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f066002ff60 0x7f0660032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f065c0096a0 tx=0x7f065c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:25.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.058+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06640095f0 con 0x7f06700f6510 2023-12-08T10:10:25.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.264+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0670045ad0 con 0x7f06700f6510 2023-12-08T10:10:25.267 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.265+0000 7f066d7fa700 1 -- 172.21.15.170:0/8021075 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f066401db70 con 0x7f06700f6510 2023-12-08T10:10:25.267 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:25.267 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:25.268 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f066002ff60 msgr2=0x7f0660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f066002ff60 0x7f0660032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f065c0096a0 tx=0x7f065c006b40).stop 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 msgr2=0x7f067006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f067006b9a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f0664013510 tx=0x7f066400b040).stop 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 shutdown_connections 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f066002ff60 0x7f0660032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 --2- 172.21.15.170:0/8021075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06700f6510 0x7f067006b9a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 >> 172.21.15.170:0/8021075 conn(0x7f06700f1c70 msgr2=0x7f06700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 shutdown_connections 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:25.266+0000 7f0676f1f700 1 -- 172.21.15.170:0/8021075 wait complete. 2023-12-08T10:10:25.269 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:26.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:24.342359+0000 mgr.smithi023.tgiigg (mgr.14164) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:26.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:25.266437+0000 mon.smithi023 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.170:0/8021075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:26.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:26.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 -- 172.21.15.170:0/685983201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 msgr2=0x7f24a00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 --2- 172.21.15.170:0/685983201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a00f66c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2490009fb0 tx=0x7f2490009b30).stop 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 -- 172.21.15.170:0/685983201 shutdown_connections 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 --2- 172.21.15.170:0/685983201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a00f66c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 -- 172.21.15.170:0/685983201 >> 172.21.15.170:0/685983201 conn(0x7f24a00f1aa0 msgr2=0x7f24a00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 -- 172.21.15.170:0/685983201 shutdown_connections 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 -- 172.21.15.170:0/685983201 wait complete. 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 Processor -- start 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.862+0000 7f249f59e700 1 -- start start 2023-12-08T10:10:27.865 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249f59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:27.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249f59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a0069e20 con 0x7f24a00f62a0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54334/0 (socket says 172.21.15.170:54334) 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249e59c700 1 -- 172.21.15.170:0/2363895445 learned_addr learned my addr 172.21.15.170:0/2363895445 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249e59c700 1 -- 172.21.15.170:0/2363895445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2490005040 con 0x7f24a00f62a0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.863+0000 7f249e59c700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a006b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2490013510 tx=0x7f249000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2490009320 con 0x7f24a00f62a0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a0069fc0 con 0x7f24a00f62a0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2490013f00 con 0x7f24a00f62a0 2023-12-08T10:10:27.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2490014dc0 con 0x7f24a00f62a0 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a006a450 con 0x7f24a00f62a0 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24900095f0 con 0x7f24a00f62a0 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.864+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a0059100 con 0x7f24a00f62a0 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.865+0000 7f248effd700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f248802ff60 0x7f2488032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.865+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2490041080 con 0x7f24a00f62a0 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.865+0000 7f249dd9b700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f248802ff60 0x7f2488032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:27.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.865+0000 7f249dd9b700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f248802ff60 0x7f2488032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f24940096a0 tx=0x7f2494006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:27.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:27.870+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24900268c0 con 0x7f24a00f62a0 2023-12-08T10:10:28.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.072+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24a00fbce0 con 0x7f24a00f62a0 2023-12-08T10:10:28.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.073+0000 7f248effd700 1 -- 172.21.15.170:0/2363895445 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2490010230 con 0x7f24a00f62a0 2023-12-08T10:10:28.078 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:28.078 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.074+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f248802ff60 msgr2=0x7f2488032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f248802ff60 0x7f2488032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f24940096a0 tx=0x7f2494006b40).stop 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 msgr2=0x7f24a006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a006b9a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f2490013510 tx=0x7f249000b040).stop 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 shutdown_connections 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f248802ff60 0x7f2488032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 --2- 172.21.15.170:0/2363895445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24a00f62a0 0x7f24a006b9a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 >> 172.21.15.170:0/2363895445 conn(0x7f24a00f1aa0 msgr2=0x7f24a00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:28.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 shutdown_connections 2023-12-08T10:10:28.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:28.075+0000 7f249f59e700 1 -- 172.21.15.170:0/2363895445 wait complete. 2023-12-08T10:10:28.080 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:28.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:26.342590+0000 mgr.smithi023.tgiigg (mgr.14164) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:29.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:28 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:28.074636+0000 mon.smithi023 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.170:0/2363895445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:29.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:29.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:28.342864+0000 mgr.smithi023.tgiigg (mgr.14164) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:30.552 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 -- 172.21.15.170:0/2044402796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 msgr2=0x7f7e100f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 --2- 172.21.15.170:0/2044402796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e100f4fa0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f7e00009fb0 tx=0x7f7e00009b30).stop 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 -- 172.21.15.170:0/2044402796 shutdown_connections 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 --2- 172.21.15.170:0/2044402796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e100f4fa0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 -- 172.21.15.170:0/2044402796 >> 172.21.15.170:0/2044402796 conn(0x7f7e100f02e0 msgr2=0x7f7e100f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 -- 172.21.15.170:0/2044402796 shutdown_connections 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.550+0000 7f7e17728700 1 -- 172.21.15.170:0/2044402796 wait complete. 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e17728700 1 Processor -- start 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e17728700 1 -- start start 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e17728700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e10106cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e17728700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e101071a0 con 0x7f7e100f4b80 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e154c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e10106cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e154c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e10106cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54352/0 (socket says 172.21.15.170:54352) 2023-12-08T10:10:30.553 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.551+0000 7f7e154c4700 1 -- 172.21.15.170:0/2543335435 learned_addr learned my addr 172.21.15.170:0/2543335435 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:30.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e154c4700 1 -- 172.21.15.170:0/2543335435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00005040 con 0x7f7e100f4b80 2023-12-08T10:10:30.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e154c4700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e10106cb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7e00013510 tx=0x7f7e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:30.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e00009320 con 0x7f7e100f4b80 2023-12-08T10:10:30.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e101073a0 con 0x7f7e100f4b80 2023-12-08T10:10:30.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e00013f00 con 0x7f7e100f4b80 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e00014d90 con 0x7f7e100f4b80 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.552+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e10107830 con 0x7f7e100f4b80 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.553+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e000263a0 con 0x7f7e100f4b80 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.553+0000 7f7e05ffb700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7dfc02ff60 0x7f7dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.553+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e00041000 con 0x7f7e100f4b80 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.553+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10045ad0 con 0x7f7e100f4b80 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.553+0000 7f7e14cc3700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7dfc02ff60 0x7f7dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:30.556 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.554+0000 7f7e14cc3700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7dfc02ff60 0x7f7dfc032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7e0c0096a0 tx=0x7f7e0c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:30.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.558+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e00012920 con 0x7f7e100f4b80 2023-12-08T10:10:30.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.768+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e100fa5d0 con 0x7f7e100f4b80 2023-12-08T10:10:30.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.769+0000 7f7e05ffb700 1 -- 172.21.15.170:0/2543335435 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e0002b020 con 0x7f7e100f4b80 2023-12-08T10:10:30.771 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:30.771 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:30.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.770+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7dfc02ff60 msgr2=0x7f7dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:30.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.770+0000 7f7e17728700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7dfc02ff60 0x7f7dfc032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f7e0c0096a0 tx=0x7f7e0c006b40).stop 2023-12-08T10:10:30.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.770+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 msgr2=0x7f7e10106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:30.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.770+0000 7f7e17728700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e10106cb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f7e00013510 tx=0x7f7e0000b040).stop 2023-12-08T10:10:30.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.771+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 shutdown_connections 2023-12-08T10:10:30.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.771+0000 7f7e17728700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7dfc02ff60 0x7f7dfc032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:30.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.771+0000 7f7e17728700 1 --2- 172.21.15.170:0/2543335435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e100f4b80 0x7f7e10106cb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:30.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.771+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 >> 172.21.15.170:0/2543335435 conn(0x7f7e100f02e0 msgr2=0x7f7e100f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:30.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.771+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 shutdown_connections 2023-12-08T10:10:30.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:30.771+0000 7f7e17728700 1 -- 172.21.15.170:0/2543335435 wait complete. 2023-12-08T10:10:30.773 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:31.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:30.769525+0000 mon.smithi023 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.170:0/2543335435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:32.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:32.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:30.343221+0000 mgr.smithi023.tgiigg (mgr.14164) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 -- 172.21.15.170:0/198577257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 msgr2=0x7f0a480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 --2- 172.21.15.170:0/198577257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a480f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f0a38009fb0 tx=0x7f0a38009b30).stop 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 -- 172.21.15.170:0/198577257 shutdown_connections 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 --2- 172.21.15.170:0/198577257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a480f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 -- 172.21.15.170:0/198577257 >> 172.21.15.170:0/198577257 conn(0x7f0a480f09a0 msgr2=0x7f0a480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 -- 172.21.15.170:0/198577257 shutdown_connections 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 -- 172.21.15.170:0/198577257 wait complete. 2023-12-08T10:10:33.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.318+0000 7f0a47fff700 1 Processor -- start 2023-12-08T10:10:33.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a47fff700 1 -- start start 2023-12-08T10:10:33.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a481053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:33.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a47fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a481058a0 con 0x7f0a480f51a0 2023-12-08T10:10:33.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a46ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a481053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:33.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a46ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a481053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:43502/0 (socket says 172.21.15.170:43502) 2023-12-08T10:10:33.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a46ffd700 1 -- 172.21.15.170:0/3125969591 learned_addr learned my addr 172.21.15.170:0/3125969591 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:33.322 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.319+0000 7f0a46ffd700 1 -- 172.21.15.170:0/3125969591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38005040 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.320+0000 7f0a46ffd700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a481053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0a38013510 tx=0x7f0a3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.320+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a38009320 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.320+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a48105aa0 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.320+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a38013f00 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.320+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a38014dc0 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.320+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a48105f30 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.321+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a3801d3d0 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.321+0000 7f0a2f7fe700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a3002ff60 0x7f0a30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.321+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a380410b0 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.321+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a48045ad0 con 0x7f0a480f51a0 2023-12-08T10:10:33.323 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.321+0000 7f0a467fc700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a3002ff60 0x7f0a30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:33.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.322+0000 7f0a467fc700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a3002ff60 0x7f0a30032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0a3c0096a0 tx=0x7f0a3c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:33.327 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.326+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a38012920 con 0x7f0a480f51a0 2023-12-08T10:10:33.534 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.533+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a48106c50 con 0x7f0a480f51a0 2023-12-08T10:10:33.535 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.534+0000 7f0a2f7fe700 1 -- 172.21.15.170:0/3125969591 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a3802b020 con 0x7f0a480f51a0 2023-12-08T10:10:33.535 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:33.535 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:33.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a3002ff60 msgr2=0x7f0a30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:33.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a3002ff60 0x7f0a30032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f0a3c0096a0 tx=0x7f0a3c006b40).stop 2023-12-08T10:10:33.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 msgr2=0x7f0a481053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:33.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a481053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f0a38013510 tx=0x7f0a3800b040).stop 2023-12-08T10:10:33.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 shutdown_connections 2023-12-08T10:10:33.537 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0a3002ff60 0x7f0a30032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:33.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 --2- 172.21.15.170:0/3125969591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a480f51a0 0x7f0a481053b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:33.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 >> 172.21.15.170:0/3125969591 conn(0x7f0a480f09a0 msgr2=0x7f0a480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:33.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 shutdown_connections 2023-12-08T10:10:33.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:33.535+0000 7f0a47fff700 1 -- 172.21.15.170:0/3125969591 wait complete. 2023-12-08T10:10:33.539 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:34.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:32.343489+0000 mgr.smithi023.tgiigg (mgr.14164) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:34.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:33.534367+0000 mon.smithi023 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.170:0/3125969591' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:34.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:34.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:36.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.074+0000 7ff427872700 1 -- 172.21.15.170:0/2157667163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4200f6510 msgr2=0x7ff4200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.074+0000 7ff427872700 1 --2- 172.21.15.170:0/2157667163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4200f6510 0x7ff4200f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7ff410009fb0 tx=0x7ff410009b30).stop 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.074+0000 7ff427872700 1 -- 172.21.15.170:0/2157667163 shutdown_connections 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.074+0000 7ff427872700 1 --2- 172.21.15.170:0/2157667163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4200f6510 0x7ff4200f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.074+0000 7ff427872700 1 -- 172.21.15.170:0/2157667163 >> 172.21.15.170:0/2157667163 conn(0x7ff4200f1c70 msgr2=0x7ff4200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.075+0000 7ff427872700 1 -- 172.21.15.170:0/2157667163 shutdown_connections 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.075+0000 7ff427872700 1 -- 172.21.15.170:0/2157667163 wait complete. 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.075+0000 7ff427872700 1 Processor -- start 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.075+0000 7ff427872700 1 -- start start 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.075+0000 7ff427872700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 0x7ff420069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:36.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.075+0000 7ff427872700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42006bd80 con 0x7ff42006b960 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff42560e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 0x7ff420069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff42560e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 0x7ff420069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:43516/0 (socket says 172.21.15.170:43516) 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff42560e700 1 -- 172.21.15.170:0/713404934 learned_addr learned my addr 172.21.15.170:0/713404934 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff42560e700 1 -- 172.21.15.170:0/713404934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410005040 con 0x7ff42006b960 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff42560e700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 0x7ff420069e20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7ff410009f80 tx=0x7ff410009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4100136c0 con 0x7ff42006b960 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42006a310 con 0x7ff42006b960 2023-12-08T10:10:36.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4100095f0 con 0x7ff42006b960 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff410014420 con 0x7ff42006b960 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.076+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42006a800 con 0x7ff42006b960 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.077+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff410025060 con 0x7ff42006b960 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.077+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420059100 con 0x7ff42006b960 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.077+0000 7ff415ffb700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff40c02ff60 0x7ff40c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.077+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff410040d80 con 0x7ff42006b960 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.078+0000 7ff424e0d700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff40c02ff60 0x7ff40c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:36.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.078+0000 7ff424e0d700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff40c02ff60 0x7ff40c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff41c0096a0 tx=0x7ff41c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:36.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.082+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff41001cb60 con 0x7ff42006b960 2023-12-08T10:10:36.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:34.343849+0000 mgr.smithi023.tgiigg (mgr.14164) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:36.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.290+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff42006abf0 con 0x7ff42006b960 2023-12-08T10:10:36.292 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.291+0000 7ff415ffb700 1 -- 172.21.15.170:0/713404934 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff410042070 con 0x7ff42006b960 2023-12-08T10:10:36.292 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:36.292 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff40c02ff60 msgr2=0x7ff40c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff40c02ff60 0x7ff40c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff41c0096a0 tx=0x7ff41c006b40).stop 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 msgr2=0x7ff420069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 0x7ff420069e20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7ff410009f80 tx=0x7ff410009b30).stop 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 shutdown_connections 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff40c02ff60 0x7ff40c032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 --2- 172.21.15.170:0/713404934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff42006b960 0x7ff420069e20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 >> 172.21.15.170:0/713404934 conn(0x7ff4200f1c70 msgr2=0x7ff4200f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:36.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 shutdown_connections 2023-12-08T10:10:36.295 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:36.293+0000 7ff427872700 1 -- 172.21.15.170:0/713404934 wait complete. 2023-12-08T10:10:36.295 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:37.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:36 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:36.291585+0000 mon.smithi023 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.170:0/713404934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:37.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:37.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:38.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:36.344261+0000 mgr.smithi023.tgiigg (mgr.14164) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:38.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.899+0000 7f652eef1700 1 -- 172.21.15.170:0/3811393878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 msgr2=0x7f65280f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:38.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.899+0000 7f652eef1700 1 --2- 172.21.15.170:0/3811393878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65280f90f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f651c009fb0 tx=0x7f651c009b30).stop 2023-12-08T10:10:38.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.899+0000 7f652eef1700 1 -- 172.21.15.170:0/3811393878 shutdown_connections 2023-12-08T10:10:38.901 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.899+0000 7f652eef1700 1 --2- 172.21.15.170:0/3811393878 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65280f90f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:38.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.899+0000 7f652eef1700 1 -- 172.21.15.170:0/3811393878 >> 172.21.15.170:0/3811393878 conn(0x7f65280f0b70 msgr2=0x7f65280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:38.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.900+0000 7f652eef1700 1 -- 172.21.15.170:0/3811393878 shutdown_connections 2023-12-08T10:10:38.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.900+0000 7f652eef1700 1 -- 172.21.15.170:0/3811393878 wait complete. 2023-12-08T10:10:38.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.900+0000 7f652eef1700 1 Processor -- start 2023-12-08T10:10:38.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.900+0000 7f652eef1700 1 -- start start 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.900+0000 7f652eef1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65281066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.900+0000 7f652eef1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6528106bd0 con 0x7f6528060280 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652cc8d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65281066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652cc8d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65281066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:43542/0 (socket says 172.21.15.170:43542) 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652cc8d700 1 -- 172.21.15.170:0/2000490973 learned_addr learned my addr 172.21.15.170:0/2000490973 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652cc8d700 1 -- 172.21.15.170:0/2000490973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c005040 con 0x7f6528060280 2023-12-08T10:10:38.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652cc8d700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65281066e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f651c013510 tx=0x7f651c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:38.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c009320 con 0x7f6528060280 2023-12-08T10:10:38.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f651c013f00 con 0x7f6528060280 2023-12-08T10:10:38.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6528106dd0 con 0x7f6528060280 2023-12-08T10:10:38.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651c014dc0 con 0x7f6528060280 2023-12-08T10:10:38.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.901+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6528107260 con 0x7f6528060280 2023-12-08T10:10:38.904 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.902+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6528045ad0 con 0x7f6528060280 2023-12-08T10:10:38.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.902+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f651c014420 con 0x7f6528060280 2023-12-08T10:10:38.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.902+0000 7f65257fa700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f651802ff60 0x7f6518032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:38.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.902+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f651c041030 con 0x7f6528060280 2023-12-08T10:10:38.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.903+0000 7f6527fff700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f651802ff60 0x7f6518032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:38.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.903+0000 7f6527fff700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f651802ff60 0x7f6518032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f65140096a0 tx=0x7f6514006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:38.908 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:38.907+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f651c0095f0 con 0x7f6528060280 2023-12-08T10:10:39.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.116+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6528108110 con 0x7f6528060280 2023-12-08T10:10:39.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.117+0000 7f65257fa700 1 -- 172.21.15.170:0/2000490973 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f651c01cb10 con 0x7f6528060280 2023-12-08T10:10:39.118 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:39.119 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.118+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f651802ff60 msgr2=0x7f6518032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.118+0000 7f652eef1700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f651802ff60 0x7f6518032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f65140096a0 tx=0x7f6514006b40).stop 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.118+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 msgr2=0x7f65281066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.118+0000 7f652eef1700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65281066e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f651c013510 tx=0x7f651c00b040).stop 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.119+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 shutdown_connections 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.119+0000 7f652eef1700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f651802ff60 0x7f6518032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.119+0000 7f652eef1700 1 --2- 172.21.15.170:0/2000490973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6528060280 0x7f65281066e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.119+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 >> 172.21.15.170:0/2000490973 conn(0x7f65280f0b70 msgr2=0x7f65280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.119+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 shutdown_connections 2023-12-08T10:10:39.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:39.119+0000 7f652eef1700 1 -- 172.21.15.170:0/2000490973 wait complete. 2023-12-08T10:10:39.121 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:40.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:38.344558+0000 mgr.smithi023.tgiigg (mgr.14164) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:40.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:39.117523+0000 mon.smithi023 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.170:0/2000490973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:40.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:40.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:41.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.676+0000 7f25e6f91700 1 -- 172.21.15.170:0/3244520247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 msgr2=0x7f25e00f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.676+0000 7f25e6f91700 1 --2- 172.21.15.170:0/3244520247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f8fc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f25d4009fb0 tx=0x7f25d4009b30).stop 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.676+0000 7f25e6f91700 1 -- 172.21.15.170:0/3244520247 shutdown_connections 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.676+0000 7f25e6f91700 1 --2- 172.21.15.170:0/3244520247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f8fc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.676+0000 7f25e6f91700 1 -- 172.21.15.170:0/3244520247 >> 172.21.15.170:0/3244520247 conn(0x7f25e00f09a0 msgr2=0x7f25e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.677+0000 7f25e6f91700 1 -- 172.21.15.170:0/3244520247 shutdown_connections 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.677+0000 7f25e6f91700 1 -- 172.21.15.170:0/3244520247 wait complete. 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.677+0000 7f25e6f91700 1 Processor -- start 2023-12-08T10:10:41.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.677+0000 7f25e6f91700 1 -- start start 2023-12-08T10:10:41.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.677+0000 7f25e6f91700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.677+0000 7f25e6f91700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e00f4ec0 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e5f8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e5f8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:43564/0 (socket says 172.21.15.170:43564) 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e5f8f700 1 -- 172.21.15.170:0/52117883 learned_addr learned my addr 172.21.15.170:0/52117883 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e5f8f700 1 -- 172.21.15.170:0/52117883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d4005040 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e5f8f700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f6a40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f25d4013510 tx=0x7f25d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d4009320 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e00f5060 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25d4013f00 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25d4014d90 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.678+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e00f54f0 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.679+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e0059100 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.679+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25d40095f0 con 0x7f25e0060140 2023-12-08T10:10:41.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.679+0000 7f25d27fc700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f25cc02ff60 0x7f25cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:41.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.679+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25d4040ff0 con 0x7f25e0060140 2023-12-08T10:10:41.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.680+0000 7f25e578e700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f25cc02ff60 0x7f25cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:41.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.680+0000 7f25e578e700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f25cc02ff60 0x7f25cc032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f25dc0096a0 tx=0x7f25dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:41.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.684+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25d4014420 con 0x7f25e0060140 2023-12-08T10:10:41.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.890+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25e00fbce0 con 0x7f25e0060140 2023-12-08T10:10:41.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.891+0000 7f25d27fc700 1 -- 172.21.15.170:0/52117883 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25d401d7f0 con 0x7f25e0060140 2023-12-08T10:10:41.893 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:41.893 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f25cc02ff60 msgr2=0x7f25cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f25cc02ff60 0x7f25cc032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f25dc0096a0 tx=0x7f25dc006b40).stop 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 msgr2=0x7f25e00f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f6a40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f25d4013510 tx=0x7f25d400b040).stop 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 shutdown_connections 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f25cc02ff60 0x7f25cc032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:41.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 --2- 172.21.15.170:0/52117883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25e0060140 0x7f25e00f6a40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:41.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 >> 172.21.15.170:0/52117883 conn(0x7f25e00f09a0 msgr2=0x7f25e00f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:41.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 shutdown_connections 2023-12-08T10:10:41.895 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:41.893+0000 7f25e6f91700 1 -- 172.21.15.170:0/52117883 wait complete. 2023-12-08T10:10:41.895 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:42.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:40.344910+0000 mgr.smithi023.tgiigg (mgr.14164) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:42.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:41.891920+0000 mon.smithi023 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.170:0/52117883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:43.278 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:43.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:42.345251+0000 mgr.smithi023.tgiigg (mgr.14164) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:44.393 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 -- 172.21.15.170:0/3441651910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 msgr2=0x7f52a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 --2- 172.21.15.170:0/3441651910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a40f55c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f5298009fb0 tx=0x7f5298009b30).stop 2023-12-08T10:10:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 -- 172.21.15.170:0/3441651910 shutdown_connections 2023-12-08T10:10:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 --2- 172.21.15.170:0/3441651910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a40f55c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 -- 172.21.15.170:0/3441651910 >> 172.21.15.170:0/3441651910 conn(0x7f52a40f09a0 msgr2=0x7f52a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 -- 172.21.15.170:0/3441651910 shutdown_connections 2023-12-08T10:10:44.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.392+0000 7f52a899e700 1 -- 172.21.15.170:0/3441651910 wait complete. 2023-12-08T10:10:44.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a899e700 1 Processor -- start 2023-12-08T10:10:44.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a899e700 1 -- start start 2023-12-08T10:10:44.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a899e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a899e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4060240 con 0x7f52a40f51a0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39432/0 (socket says 172.21.15.170:39432) 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.393+0000 7f52a37fe700 1 -- 172.21.15.170:0/534758539 learned_addr learned my addr 172.21.15.170:0/534758539 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f52a37fe700 1 -- 172.21.15.170:0/534758539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5298005040 con 0x7f52a40f51a0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f52a37fe700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a41091b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f5298013510 tx=0x7f529800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5298009320 con 0x7f52a40f51a0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a4060440 con 0x7f52a40f51a0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5298013f00 con 0x7f52a40f51a0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5298014dc0 con 0x7f52a40f51a0 2023-12-08T10:10:44.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.394+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a4109860 con 0x7f52a40f51a0 2023-12-08T10:10:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.395+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f529801d3d0 con 0x7f52a40f51a0 2023-12-08T10:10:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.395+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a4045ad0 con 0x7f52a40f51a0 2023-12-08T10:10:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.395+0000 7f528bfff700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f528c02ff60 0x7f528c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.395+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5298041310 con 0x7f52a40f51a0 2023-12-08T10:10:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.395+0000 7f52a2ffd700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f528c02ff60 0x7f528c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:44.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.396+0000 7f52a2ffd700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f528c02ff60 0x7f528c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f52940096a0 tx=0x7f5294006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:44.401 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.400+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52980126f0 con 0x7f52a40f51a0 2023-12-08T10:10:44.607 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.606+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52a410a080 con 0x7f52a40f51a0 2023-12-08T10:10:44.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.607+0000 7f528bfff700 1 -- 172.21.15.170:0/534758539 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5298012330 con 0x7f52a40f51a0 2023-12-08T10:10:44.608 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:44.608 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f528c02ff60 msgr2=0x7f528c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f528c02ff60 0x7f528c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f52940096a0 tx=0x7f5294006b40).stop 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 msgr2=0x7f52a41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a41091b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f5298013510 tx=0x7f529800b040).stop 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 shutdown_connections 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f528c02ff60 0x7f528c032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:44.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 --2- 172.21.15.170:0/534758539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f52a40f51a0 0x7f52a41091b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:44.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 >> 172.21.15.170:0/534758539 conn(0x7f52a40f09a0 msgr2=0x7f52a40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:44.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 shutdown_connections 2023-12-08T10:10:44.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:44.609+0000 7f52a899e700 1 -- 172.21.15.170:0/534758539 wait complete. 2023-12-08T10:10:44.611 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:45.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:44 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:44.607665+0000 mon.smithi023 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.170:0/534758539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:45.974 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:45.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:44.345516+0000 mgr.smithi023.tgiigg (mgr.14164) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:47.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 -- 172.21.15.170:0/3070757883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 msgr2=0x7f5c980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 --2- 172.21.15.170:0/3070757883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c980f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f5c88009fb0 tx=0x7f5c88009b30).stop 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 -- 172.21.15.170:0/3070757883 shutdown_connections 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 --2- 172.21.15.170:0/3070757883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c980f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 -- 172.21.15.170:0/3070757883 >> 172.21.15.170:0/3070757883 conn(0x7f5c980f0b70 msgr2=0x7f5c980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 -- 172.21.15.170:0/3070757883 shutdown_connections 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.114+0000 7f5c9fb02700 1 -- 172.21.15.170:0/3070757883 wait complete. 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9fb02700 1 Processor -- start 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9fb02700 1 -- start start 2023-12-08T10:10:47.118 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9fb02700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c98105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9fb02700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98105b20 con 0x7f5c980f5410 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9d89e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c98105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9d89e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c98105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39448/0 (socket says 172.21.15.170:39448) 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.115+0000 7f5c9d89e700 1 -- 172.21.15.170:0/2792452844 learned_addr learned my addr 172.21.15.170:0/2792452844 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c9d89e700 1 -- 172.21.15.170:0/2792452844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88005040 con 0x7f5c980f5410 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c9d89e700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c98105630 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5c88013510 tx=0x7f5c8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88009320 con 0x7f5c980f5410 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c98105d20 con 0x7f5c980f5410 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c88013f00 con 0x7f5c980f5410 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88014d90 con 0x7f5c980f5410 2023-12-08T10:10:47.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.116+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c981061b0 con 0x7f5c980f5410 2023-12-08T10:10:47.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.117+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c880263a0 con 0x7f5c980f5410 2023-12-08T10:10:47.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.117+0000 7f5c8e7fc700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c8402ff60 0x7f5c84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:47.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.117+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c880413a0 con 0x7f5c980f5410 2023-12-08T10:10:47.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.117+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c98045ad0 con 0x7f5c980f5410 2023-12-08T10:10:47.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.117+0000 7f5c9d09d700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c8402ff60 0x7f5c84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:47.121 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.118+0000 7f5c9d09d700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c8402ff60 0x7f5c84032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5c940096a0 tx=0x7f5c94006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:47.123 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.122+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c88012510 con 0x7f5c980f5410 2023-12-08T10:10:47.329 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.327+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c980faf50 con 0x7f5c980f5410 2023-12-08T10:10:47.330 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.329+0000 7f5c8e7fc700 1 -- 172.21.15.170:0/2792452844 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c8801ccb0 con 0x7f5c980f5410 2023-12-08T10:10:47.331 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:47.331 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:47.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c8402ff60 msgr2=0x7f5c84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:47.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c8402ff60 0x7f5c84032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5c940096a0 tx=0x7f5c94006b40).stop 2023-12-08T10:10:47.332 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 msgr2=0x7f5c98105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c98105630 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5c88013510 tx=0x7f5c8800b040).stop 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 shutdown_connections 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c8402ff60 0x7f5c84032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 --2- 172.21.15.170:0/2792452844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c980f5410 0x7f5c98105630 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 >> 172.21.15.170:0/2792452844 conn(0x7f5c980f0b70 msgr2=0x7f5c980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 shutdown_connections 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:47.330+0000 7f5c9fb02700 1 -- 172.21.15.170:0/2792452844 wait complete. 2023-12-08T10:10:47.333 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:48.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:46.345867+0000 mgr.smithi023.tgiigg (mgr.14164) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:48.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:47.329452+0000 mon.smithi023 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.170:0/2792452844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:48.715 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:48.716 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.864+0000 7f696951a700 1 -- 172.21.15.170:0/2060056824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 msgr2=0x7f69640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.864+0000 7f696951a700 1 --2- 172.21.15.170:0/2060056824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69640f55c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f6954009fb0 tx=0x7f6954009b30).stop 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 -- 172.21.15.170:0/2060056824 shutdown_connections 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 --2- 172.21.15.170:0/2060056824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69640f55c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 -- 172.21.15.170:0/2060056824 >> 172.21.15.170:0/2060056824 conn(0x7f69640f09a0 msgr2=0x7f69640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 -- 172.21.15.170:0/2060056824 shutdown_connections 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 -- 172.21.15.170:0/2060056824 wait complete. 2023-12-08T10:10:49.868 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 Processor -- start 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 -- start start 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69641091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.865+0000 7f696951a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6964060240 con 0x7f69640f51a0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6963fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69641091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6963fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69641091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:39452/0 (socket says 172.21.15.170:39452) 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6963fff700 1 -- 172.21.15.170:0/2038734225 learned_addr learned my addr 172.21.15.170:0/2038734225 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6963fff700 1 -- 172.21.15.170:0/2038734225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954005040 con 0x7f69640f51a0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6963fff700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69641091b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f6954013510 tx=0x7f695400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6954009320 con 0x7f69640f51a0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6964060440 con 0x7f69640f51a0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6954013f00 con 0x7f69640f51a0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6954014dc0 con 0x7f69640f51a0 2023-12-08T10:10:49.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.866+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6964109860 con 0x7f69640f51a0 2023-12-08T10:10:49.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.867+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69540095f0 con 0x7f69640f51a0 2023-12-08T10:10:49.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.867+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6964045ad0 con 0x7f69640f51a0 2023-12-08T10:10:49.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.867+0000 7f6960ff9700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f694c02ff60 0x7f694c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:49.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.867+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6954041020 con 0x7f69640f51a0 2023-12-08T10:10:49.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.867+0000 7f69637fe700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f694c02ff60 0x7f694c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:49.871 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.868+0000 7f69637fe700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f694c02ff60 0x7f694c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f69580096a0 tx=0x7f6958006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:49.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:49.872+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6954014420 con 0x7f69640f51a0 2023-12-08T10:10:50.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.079+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f696410a070 con 0x7f69640f51a0 2023-12-08T10:10:50.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.080+0000 7f6960ff9700 1 -- 172.21.15.170:0/2038734225 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f695401c920 con 0x7f69640f51a0 2023-12-08T10:10:50.082 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:50.082 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:50.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f694c02ff60 msgr2=0x7f694c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:50.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f694c02ff60 0x7f694c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f69580096a0 tx=0x7f6958006b40).stop 2023-12-08T10:10:50.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 msgr2=0x7f69641091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69641091b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f6954013510 tx=0x7f695400b040).stop 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 shutdown_connections 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f694c02ff60 0x7f694c032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 --2- 172.21.15.170:0/2038734225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69640f51a0 0x7f69641091b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 >> 172.21.15.170:0/2038734225 conn(0x7f69640f09a0 msgr2=0x7f69640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 shutdown_connections 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:50.082+0000 7f696951a700 1 -- 172.21.15.170:0/2038734225 wait complete. 2023-12-08T10:10:50.084 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:48.346235+0000 mgr.smithi023.tgiigg (mgr.14164) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:51.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:50.080852+0000 mon.smithi023 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.170:0/2038734225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:51.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:51.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:52.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:50.346586+0000 mgr.smithi023.tgiigg (mgr.14164) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 -- 172.21.15.170:0/4281651892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 msgr2=0x7f9a880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 --2- 172.21.15.170:0/4281651892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a880f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f9a7c009fb0 tx=0x7f9a7c009b30).stop 2023-12-08T10:10:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 -- 172.21.15.170:0/4281651892 shutdown_connections 2023-12-08T10:10:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 --2- 172.21.15.170:0/4281651892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a880f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 -- 172.21.15.170:0/4281651892 >> 172.21.15.170:0/4281651892 conn(0x7f9a880f0b70 msgr2=0x7f9a880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 -- 172.21.15.170:0/4281651892 shutdown_connections 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.632+0000 7f9a8ebee700 1 -- 172.21.15.170:0/4281651892 wait complete. 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8ebee700 1 Processor -- start 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8ebee700 1 -- start start 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8ebee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a88109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8ebee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88060220 con 0x7f9a880f5410 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8c98a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a88109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8c98a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a88109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52624/0 (socket says 172.21.15.170:52624) 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.633+0000 7f9a8c98a700 1 -- 172.21.15.170:0/2394052671 learned_addr learned my addr 172.21.15.170:0/2394052671 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a8c98a700 1 -- 172.21.15.170:0/2394052671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c005040 con 0x7f9a880f5410 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a8c98a700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a88109450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f9a7c013510 tx=0x7f9a7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7c009320 con 0x7f9a880f5410 2023-12-08T10:10:52.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a88060420 con 0x7f9a880f5410 2023-12-08T10:10:52.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a7c013f00 con 0x7f9a880f5410 2023-12-08T10:10:52.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a7c014d90 con 0x7f9a880f5410 2023-12-08T10:10:52.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.634+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a88109b00 con 0x7f9a880f5410 2023-12-08T10:10:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.635+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a7c014420 con 0x7f9a880f5410 2023-12-08T10:10:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.635+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a88059100 con 0x7f9a880f5410 2023-12-08T10:10:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.635+0000 7f9a857fa700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9a7802ff60 0x7f9a78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.635+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a7c041000 con 0x7f9a880f5410 2023-12-08T10:10:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.635+0000 7f9a87fff700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9a7802ff60 0x7f9a78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:52.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.636+0000 7f9a87fff700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9a7802ff60 0x7f9a78032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9a740096a0 tx=0x7f9a74006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:52.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.640+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a7c0095f0 con 0x7f9a880f5410 2023-12-08T10:10:52.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.844+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a88045ad0 con 0x7f9a880f5410 2023-12-08T10:10:52.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.844+0000 7f9a857fa700 1 -- 172.21.15.170:0/2394052671 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a7c01db70 con 0x7f9a880f5410 2023-12-08T10:10:52.847 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:52.847 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:52.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9a7802ff60 msgr2=0x7f9a78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9a7802ff60 0x7f9a78032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f9a740096a0 tx=0x7f9a74006b40).stop 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 msgr2=0x7f9a88109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a88109450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f9a7c013510 tx=0x7f9a7c00b040).stop 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 shutdown_connections 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9a7802ff60 0x7f9a78032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 --2- 172.21.15.170:0/2394052671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a880f5410 0x7f9a88109450 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 >> 172.21.15.170:0/2394052671 conn(0x7f9a880f0b70 msgr2=0x7f9a880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 shutdown_connections 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:52.846+0000 7f9a8ebee700 1 -- 172.21.15.170:0/2394052671 wait complete. 2023-12-08T10:10:52.849 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:54.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:52.346880+0000 mgr.smithi023.tgiigg (mgr.14164) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:54.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:52.846326+0000 mon.smithi023 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.170:0/2394052671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:54.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:54.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:55.430 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 -- 172.21.15.170:0/866733899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 msgr2=0x7f91600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:55.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 --2- 172.21.15.170:0/866733899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f91600f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f9150009fb0 tx=0x7f9150009b30).stop 2023-12-08T10:10:55.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 -- 172.21.15.170:0/866733899 shutdown_connections 2023-12-08T10:10:55.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 --2- 172.21.15.170:0/866733899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f91600f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:55.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 -- 172.21.15.170:0/866733899 >> 172.21.15.170:0/866733899 conn(0x7f91600f0b70 msgr2=0x7f91600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:55.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 -- 172.21.15.170:0/866733899 shutdown_connections 2023-12-08T10:10:55.431 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.428+0000 7f9167d14700 1 -- 172.21.15.170:0/866733899 wait complete. 2023-12-08T10:10:55.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9167d14700 1 Processor -- start 2023-12-08T10:10:55.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9167d14700 1 -- start start 2023-12-08T10:10:55.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9167d14700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f9160105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:55.432 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9167d14700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9160105b20 con 0x7f91600f5410 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9165ab0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f9160105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9165ab0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f9160105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52642/0 (socket says 172.21.15.170:52642) 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.429+0000 7f9165ab0700 1 -- 172.21.15.170:0/1550515690 learned_addr learned my addr 172.21.15.170:0/1550515690 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f9165ab0700 1 -- 172.21.15.170:0/1550515690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150005040 con 0x7f91600f5410 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f9165ab0700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f9160105630 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f9150013510 tx=0x7f915000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9150009320 con 0x7f91600f5410 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9160105d20 con 0x7f91600f5410 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9150013f00 con 0x7f91600f5410 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9150014d90 con 0x7f91600f5410 2023-12-08T10:10:55.433 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.430+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91601061b0 con 0x7f91600f5410 2023-12-08T10:10:55.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.431+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91500263a0 con 0x7f91600f5410 2023-12-08T10:10:55.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.431+0000 7f91567fc700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f914c02ff60 0x7f914c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:55.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.431+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9150041040 con 0x7f91600f5410 2023-12-08T10:10:55.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.431+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9160045ad0 con 0x7f91600f5410 2023-12-08T10:10:55.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.431+0000 7f91652af700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f914c02ff60 0x7f914c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:55.434 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.432+0000 7f91652af700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f914c02ff60 0x7f914c032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f915c0096a0 tx=0x7f915c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:55.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.436+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9150012920 con 0x7f91600f5410 2023-12-08T10:10:55.644 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.642+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91600faf50 con 0x7f91600f5410 2023-12-08T10:10:55.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.642+0000 7f91567fc700 1 -- 172.21.15.170:0/1550515690 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f915002b020 con 0x7f91600f5410 2023-12-08T10:10:55.645 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:55.645 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:55.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f914c02ff60 msgr2=0x7f914c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f914c02ff60 0x7f914c032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f915c0096a0 tx=0x7f915c006b40).stop 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 msgr2=0x7f9160105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f9160105630 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f9150013510 tx=0x7f915000b040).stop 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 shutdown_connections 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f914c02ff60 0x7f914c032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 --2- 172.21.15.170:0/1550515690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91600f5410 0x7f9160105630 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 >> 172.21.15.170:0/1550515690 conn(0x7f91600f0b70 msgr2=0x7f91600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 shutdown_connections 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:55.644+0000 7f9167d14700 1 -- 172.21.15.170:0/1550515690 wait complete. 2023-12-08T10:10:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:56.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:54.347226+0000 mgr.smithi023.tgiigg (mgr.14164) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:56.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:55.644294+0000 mon.smithi023 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.170:0/1550515690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:57.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:57.037 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- 172.21.15.170:0/17918081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 msgr2=0x7fa53c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 --2- 172.21.15.170:0/17918081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fa538009fb0 tx=0x7fa538009b30).stop 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- 172.21.15.170:0/17918081 shutdown_connections 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 --2- 172.21.15.170:0/17918081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- 172.21.15.170:0/17918081 >> 172.21.15.170:0/17918081 conn(0x7fa53c0f0b70 msgr2=0x7fa53c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- 172.21.15.170:0/17918081 shutdown_connections 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- 172.21.15.170:0/17918081 wait complete. 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 Processor -- start 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- start start 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:58.166 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.162+0000 7fa5439b1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53c107240 con 0x7fa53c0f5410 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa54174d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa54174d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52664/0 (socket says 172.21.15.170:52664) 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa54174d700 1 -- 172.21.15.170:0/3316257032 learned_addr learned my addr 172.21.15.170:0/3316257032 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa54174d700 1 -- 172.21.15.170:0/3316257032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538005040 con 0x7fa53c0f5410 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa54174d700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c10b850 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa538014000 tx=0x7fa53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5380095f0 con 0x7fa53c0f5410 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53c060220 con 0x7fa53c0f5410 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa538007e70 con 0x7fa53c0f5410 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.163+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa538014ea0 con 0x7fa53c0f5410 2023-12-08T10:10:58.167 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.164+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53c10bd40 con 0x7fa53c0f5410 2023-12-08T10:10:58.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.164+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa538025070 con 0x7fa53c0f5410 2023-12-08T10:10:58.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.164+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53c045ad0 con 0x7fa53c0f5410 2023-12-08T10:10:58.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.165+0000 7fa531ffb700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa52802ff60 0x7fa528032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:10:58.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.165+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa538040920 con 0x7fa53c0f5410 2023-12-08T10:10:58.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.165+0000 7fa540f4c700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa52802ff60 0x7fa528032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:10:58.168 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.165+0000 7fa540f4c700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa52802ff60 0x7fa528032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa52c0096a0 tx=0x7fa52c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:10:58.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.169+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa53800f4e0 con 0x7fa53c0f5410 2023-12-08T10:10:58.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:56.347560+0000 mgr.smithi023.tgiigg (mgr.14164) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:10:58.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.374+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa53c0fae60 con 0x7fa53c0f5410 2023-12-08T10:10:58.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.376+0000 7fa531ffb700 1 -- 172.21.15.170:0/3316257032 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa53800f300 con 0x7fa53c0f5410 2023-12-08T10:10:58.377 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:10:58.378 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:10:58.379 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa52802ff60 msgr2=0x7fa528032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa52802ff60 0x7fa528032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa52c0096a0 tx=0x7fa52c006b40).stop 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 msgr2=0x7fa53c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c10b850 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa538014000 tx=0x7fa53800b040).stop 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 shutdown_connections 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa52802ff60 0x7fa528032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 --2- 172.21.15.170:0/3316257032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa53c0f5410 0x7fa53c10b850 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 >> 172.21.15.170:0/3316257032 conn(0x7fa53c0f0b70 msgr2=0x7fa53c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 shutdown_connections 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:10:58.377+0000 7fa5439b1700 1 -- 172.21.15.170:0/3316257032 wait complete. 2023-12-08T10:10:58.380 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:10:59.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:10:58.376820+0000 mon.smithi023 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.170:0/3316257032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:10:59.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:10:59.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:00.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:10:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:10:58.347875+0000 mgr.smithi023.tgiigg (mgr.14164) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:00.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 -- 172.21.15.170:0/1286312158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 msgr2=0x7fa55c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:00.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 --2- 172.21.15.170:0/1286312158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c0f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fa54c009fb0 tx=0x7fa54c009b30).stop 2023-12-08T10:11:00.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 -- 172.21.15.170:0/1286312158 shutdown_connections 2023-12-08T10:11:00.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 --2- 172.21.15.170:0/1286312158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c0f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:00.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 -- 172.21.15.170:0/1286312158 >> 172.21.15.170:0/1286312158 conn(0x7fa55c0f0b70 msgr2=0x7fa55c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:00.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 -- 172.21.15.170:0/1286312158 shutdown_connections 2023-12-08T10:11:00.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 -- 172.21.15.170:0/1286312158 wait complete. 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 Processor -- start 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.923+0000 7fa56146c700 1 -- start start 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa56146c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa56146c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c106bd0 con 0x7fa55c0f5410 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa55affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa55affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:52688/0 (socket says 172.21.15.170:52688) 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa55affd700 1 -- 172.21.15.170:0/1741505908 learned_addr learned my addr 172.21.15.170:0/1741505908 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa55affd700 1 -- 172.21.15.170:0/1741505908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c005040 con 0x7fa55c0f5410 2023-12-08T10:11:00.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.924+0000 7fa55affd700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c1066e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa54c013510 tx=0x7fa54c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.925+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa54c009320 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.925+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa54c013f00 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.925+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c106dd0 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.925+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa54c014d90 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.925+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55c107260 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.925+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa54c0263a0 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.926+0000 7fa5437fe700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa54402ff60 0x7fa544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.926+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa54c040fd0 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.926+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55c045ad0 con 0x7fa55c0f5410 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.926+0000 7fa55a7fc700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa54402ff60 0x7fa544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:00.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.926+0000 7fa55a7fc700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa54402ff60 0x7fa544032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa5500096a0 tx=0x7fa550006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:00.933 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:00.931+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa54c012920 con 0x7fa55c0f5410 2023-12-08T10:11:01.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.139+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa55c0faf50 con 0x7fa55c0f5410 2023-12-08T10:11:01.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.140+0000 7fa5437fe700 1 -- 172.21.15.170:0/1741505908 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa54c02b020 con 0x7fa55c0f5410 2023-12-08T10:11:01.142 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:01.142 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa54402ff60 msgr2=0x7fa544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa54402ff60 0x7fa544032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa5500096a0 tx=0x7fa550006b40).stop 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 msgr2=0x7fa55c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c1066e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa54c013510 tx=0x7fa54c00b040).stop 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 shutdown_connections 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa54402ff60 0x7fa544032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 --2- 172.21.15.170:0/1741505908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa55c0f5410 0x7fa55c1066e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 >> 172.21.15.170:0/1741505908 conn(0x7fa55c0f0b70 msgr2=0x7fa55c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.141+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 shutdown_connections 2023-12-08T10:11:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:01.142+0000 7fa56146c700 1 -- 172.21.15.170:0/1741505908 wait complete. 2023-12-08T10:11:01.145 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:02.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:00.348136+0000 mgr.smithi023.tgiigg (mgr.14164) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:02.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:01.141538+0000 mon.smithi023 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.170:0/1741505908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:02.556 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:02.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.696+0000 7f7bbbfff700 1 -- 172.21.15.170:0/3543866640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 msgr2=0x7f7bbc0f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.696+0000 7f7bbbfff700 1 --2- 172.21.15.170:0/3543866640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc0f8fc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f7bac009fb0 tx=0x7f7bac009b30).stop 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.696+0000 7f7bbbfff700 1 -- 172.21.15.170:0/3543866640 shutdown_connections 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.696+0000 7f7bbbfff700 1 --2- 172.21.15.170:0/3543866640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc0f8fc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.696+0000 7f7bbbfff700 1 -- 172.21.15.170:0/3543866640 >> 172.21.15.170:0/3543866640 conn(0x7f7bbc0f09a0 msgr2=0x7f7bbc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.696+0000 7f7bbbfff700 1 -- 172.21.15.170:0/3543866640 shutdown_connections 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.697+0000 7f7bbbfff700 1 -- 172.21.15.170:0/3543866640 wait complete. 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.697+0000 7f7bbbfff700 1 Processor -- start 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.697+0000 7f7bbbfff700 1 -- start start 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.697+0000 7f7bbbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.697+0000 7f7bbbfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc106a20 con 0x7f7bbc060140 2023-12-08T10:11:03.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:03.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:56700/0 (socket says 172.21.15.170:56700) 2023-12-08T10:11:03.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbaffd700 1 -- 172.21.15.170:0/477329916 learned_addr learned my addr 172.21.15.170:0/477329916 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:03.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbaffd700 1 -- 172.21.15.170:0/477329916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac005040 con 0x7f7bbc060140 2023-12-08T10:11:03.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbaffd700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc106530 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7bac013510 tx=0x7f7bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:03.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bac009320 con 0x7f7bbc060140 2023-12-08T10:11:03.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc106c20 con 0x7f7bbc060140 2023-12-08T10:11:03.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7bac013f00 con 0x7f7bbc060140 2023-12-08T10:11:03.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bac014d90 con 0x7f7bbc060140 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.698+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc1070b0 con 0x7f7bbc060140 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.699+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bac014420 con 0x7f7bbc060140 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.699+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc045ad0 con 0x7f7bbc060140 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.699+0000 7f7ba37fe700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ba402ff60 0x7f7ba4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.699+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bac0410c0 con 0x7f7bbc060140 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.699+0000 7f7bba7fc700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ba402ff60 0x7f7ba4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:03.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.700+0000 7f7bba7fc700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ba402ff60 0x7f7ba4032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7bb00096a0 tx=0x7f7bb0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:03.706 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.704+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bac0095f0 con 0x7f7bbc060140 2023-12-08T10:11:03.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.912+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bbc107f60 con 0x7f7bbc060140 2023-12-08T10:11:03.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.913+0000 7f7ba37fe700 1 -- 172.21.15.170:0/477329916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bac01db70 con 0x7f7bbc060140 2023-12-08T10:11:03.915 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:03.915 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:03.916 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ba402ff60 msgr2=0x7f7ba4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ba402ff60 0x7f7ba4032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7bb00096a0 tx=0x7f7bb0006b40).stop 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 msgr2=0x7f7bbc106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc106530 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7bac013510 tx=0x7f7bac00b040).stop 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 shutdown_connections 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7ba402ff60 0x7f7ba4032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 --2- 172.21.15.170:0/477329916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7bbc060140 0x7f7bbc106530 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 >> 172.21.15.170:0/477329916 conn(0x7f7bbc0f09a0 msgr2=0x7f7bbc0f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 shutdown_connections 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:03.914+0000 7f7bbbfff700 1 -- 172.21.15.170:0/477329916 wait complete. 2023-12-08T10:11:03.917 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:04.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:02.348395+0000 mgr.smithi023.tgiigg (mgr.14164) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:05.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:03.914406+0000 mon.smithi023 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.170:0/477329916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:05.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:05.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:06.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:04.348649+0000 mgr.smithi023.tgiigg (mgr.14164) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- 172.21.15.170:0/854258571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 msgr2=0x7f47840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 --2- 172.21.15.170:0/854258571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47840f90f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f4774009fb0 tx=0x7f4774009b30).stop 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- 172.21.15.170:0/854258571 shutdown_connections 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 --2- 172.21.15.170:0/854258571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47840f90f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- 172.21.15.170:0/854258571 >> 172.21.15.170:0/854258571 conn(0x7f47840f0b70 msgr2=0x7f47840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- 172.21.15.170:0/854258571 shutdown_connections 2023-12-08T10:11:06.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- 172.21.15.170:0/854258571 wait complete. 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 Processor -- start 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- start start 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.480+0000 7f4789cae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4784106bd0 con 0x7f4784060280 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f47837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f47837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:56718/0 (socket says 172.21.15.170:56718) 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f47837fe700 1 -- 172.21.15.170:0/637931 learned_addr learned my addr 172.21.15.170:0/637931 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f47837fe700 1 -- 172.21.15.170:0/637931 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774005040 con 0x7f4784060280 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f47837fe700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47841066e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f4774013510 tx=0x7f477400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4774009320 con 0x7f4784060280 2023-12-08T10:11:06.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4784106dd0 con 0x7f4784060280 2023-12-08T10:11:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4774013f00 con 0x7f4784060280 2023-12-08T10:11:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4774014dc0 con 0x7f4784060280 2023-12-08T10:11:06.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.481+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4784107260 con 0x7f4784060280 2023-12-08T10:11:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.482+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f477401d3d0 con 0x7f4784060280 2023-12-08T10:11:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.482+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4784045ad0 con 0x7f4784060280 2023-12-08T10:11:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.482+0000 7f476bfff700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f476c02ff60 0x7f476c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.482+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4774041050 con 0x7f4784060280 2023-12-08T10:11:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.483+0000 7f4782ffd700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f476c02ff60 0x7f476c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:06.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.483+0000 7f4782ffd700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f476c02ff60 0x7f476c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f47780096a0 tx=0x7f4778006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:06.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.487+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47740095f0 con 0x7f4784060280 2023-12-08T10:11:06.695 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.693+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47841080b0 con 0x7f4784060280 2023-12-08T10:11:06.696 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.694+0000 7f476bfff700 1 -- 172.21.15.170:0/637931 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f477401cb10 con 0x7f4784060280 2023-12-08T10:11:06.697 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:06.697 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f476c02ff60 msgr2=0x7f476c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f476c02ff60 0x7f476c032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f47780096a0 tx=0x7f4778006b40).stop 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 msgr2=0x7f47841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47841066e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f4774013510 tx=0x7f477400b040).stop 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 shutdown_connections 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f476c02ff60 0x7f476c032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 --2- 172.21.15.170:0/637931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4784060280 0x7f47841066e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 >> 172.21.15.170:0/637931 conn(0x7f47840f0b70 msgr2=0x7f47840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:06.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 shutdown_connections 2023-12-08T10:11:06.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:06.696+0000 7f4789cae700 1 -- 172.21.15.170:0/637931 wait complete. 2023-12-08T10:11:06.699 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:07.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:06.695555+0000 mon.smithi023 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.170:0/637931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:08.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:08.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:08.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:06.348996+0000 mgr.smithi023.tgiigg (mgr.14164) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:09.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 -- 172.21.15.170:0/3794960087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 msgr2=0x7fb4780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 --2- 172.21.15.170:0/3794960087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb4780f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fb46c009fb0 tx=0x7fb46c009b30).stop 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 -- 172.21.15.170:0/3794960087 shutdown_connections 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 --2- 172.21.15.170:0/3794960087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb4780f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 -- 172.21.15.170:0/3794960087 >> 172.21.15.170:0/3794960087 conn(0x7fb4780f1c70 msgr2=0x7fb4780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 -- 172.21.15.170:0/3794960087 shutdown_connections 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.278+0000 7fb47ce46700 1 -- 172.21.15.170:0/3794960087 wait complete. 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47ce46700 1 Processor -- start 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47ce46700 1 -- start start 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47ce46700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb478106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47ce46700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb478106c20 con 0x7fb4780f6510 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb478106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb478106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:56732/0 (socket says 172.21.15.170:56732) 2023-12-08T10:11:09.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47659c700 1 -- 172.21.15.170:0/2502336368 learned_addr learned my addr 172.21.15.170:0/2502336368 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:09.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47659c700 1 -- 172.21.15.170:0/2502336368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c005040 con 0x7fb4780f6510 2023-12-08T10:11:09.283 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.279+0000 7fb47659c700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb478106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb46c013510 tx=0x7fb46c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.280+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46c009320 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.280+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb478106e20 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.280+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb46c013f00 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.280+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46c014d90 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.280+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4781072b0 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.280+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb46c0095f0 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.281+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb478045ad0 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.281+0000 7fb466ffd700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb46002ff60 0x7fb460032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.281+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb46c0410c0 con 0x7fb4780f6510 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.281+0000 7fb475d9b700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb46002ff60 0x7fb460032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:09.284 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.281+0000 7fb475d9b700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb46002ff60 0x7fb460032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb4680096a0 tx=0x7fb468006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:09.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.285+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb46c014420 con 0x7fb4780f6510 2023-12-08T10:11:09.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.488+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4780fbf60 con 0x7fb4780f6510 2023-12-08T10:11:09.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb466ffd700 1 -- 172.21.15.170:0/2502336368 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb46c010230 con 0x7fb4780f6510 2023-12-08T10:11:09.491 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:09.491 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:09.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb46002ff60 msgr2=0x7fb460032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:09.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb46002ff60 0x7fb460032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb4680096a0 tx=0x7fb468006b40).stop 2023-12-08T10:11:09.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 msgr2=0x7fb478106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:09.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb478106730 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb46c013510 tx=0x7fb46c00b040).stop 2023-12-08T10:11:09.493 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 shutdown_connections 2023-12-08T10:11:09.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb46002ff60 0x7fb460032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:09.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 --2- 172.21.15.170:0/2502336368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4780f6510 0x7fb478106730 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:09.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 >> 172.21.15.170:0/2502336368 conn(0x7fb4780f1c70 msgr2=0x7fb4780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:09.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 shutdown_connections 2023-12-08T10:11:09.494 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:09.490+0000 7fb47ce46700 1 -- 172.21.15.170:0/2502336368 wait complete. 2023-12-08T10:11:09.494 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:10.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:08.349316+0000 mgr.smithi023.tgiigg (mgr.14164) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:10.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:09.490563+0000 mon.smithi023 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.170:0/2502336368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:10.894 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:10.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.020+0000 7f4cd08be700 1 -- 172.21.15.170:0/2678469094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 msgr2=0x7f4ccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.020+0000 7f4cd08be700 1 --2- 172.21.15.170:0/2678469094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc0f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f4cb4009fb0 tx=0x7f4cb4009b30).stop 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 -- 172.21.15.170:0/2678469094 shutdown_connections 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 --2- 172.21.15.170:0/2678469094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc0f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 -- 172.21.15.170:0/2678469094 >> 172.21.15.170:0/2678469094 conn(0x7f4ccc0f0b70 msgr2=0x7f4ccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 -- 172.21.15.170:0/2678469094 shutdown_connections 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 -- 172.21.15.170:0/2678469094 wait complete. 2023-12-08T10:11:12.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 Processor -- start 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 -- start start 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cd08be700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ccc060240 con 0x7f4ccc0f5410 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.021+0000 7f4cca59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cca59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:56744/0 (socket says 172.21.15.170:56744) 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cca59c700 1 -- 172.21.15.170:0/1013765794 learned_addr learned my addr 172.21.15.170:0/1013765794 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cca59c700 1 -- 172.21.15.170:0/1013765794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb4005040 con 0x7f4ccc0f5410 2023-12-08T10:11:12.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cca59c700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc109430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4cb4014000 tx=0x7f4cb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:12.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb40095f0 con 0x7f4ccc0f5410 2023-12-08T10:11:12.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ccc060440 con 0x7f4ccc0f5410 2023-12-08T10:11:12.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4cb4007e70 con 0x7f4ccc0f5410 2023-12-08T10:11:12.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb4014ea0 con 0x7f4ccc0f5410 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.022+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ccc109ae0 con 0x7f4ccc0f5410 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.023+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4cb4025070 con 0x7f4ccc0f5410 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.023+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc045ad0 con 0x7f4ccc0f5410 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.023+0000 7f4cc2ffd700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4cb802ff60 0x7f4cb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.023+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4cb4040970 con 0x7f4ccc0f5410 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.023+0000 7f4cc9d9b700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4cb802ff60 0x7f4cb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:12.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.024+0000 7f4cc9d9b700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4cb802ff60 0x7f4cb8032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4cbc0096a0 tx=0x7f4cbc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:12.030 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.029+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4cb40079c0 con 0x7f4ccc0f5410 2023-12-08T10:11:12.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:10.349655+0000 mgr.smithi023.tgiigg (mgr.14164) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:12.236 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.234+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ccc0fae60 con 0x7f4ccc0f5410 2023-12-08T10:11:12.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.235+0000 7f4cc2ffd700 1 -- 172.21.15.170:0/1013765794 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4cb400f300 con 0x7f4ccc0f5410 2023-12-08T10:11:12.237 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:12.237 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4cb802ff60 msgr2=0x7f4cb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4cb802ff60 0x7f4cb8032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4cbc0096a0 tx=0x7f4cbc006b40).stop 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 msgr2=0x7f4ccc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc109430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f4cb4014000 tx=0x7f4cb400b040).stop 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 shutdown_connections 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4cb802ff60 0x7f4cb8032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 --2- 172.21.15.170:0/1013765794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ccc0f5410 0x7f4ccc109430 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.236+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 >> 172.21.15.170:0/1013765794 conn(0x7f4ccc0f0b70 msgr2=0x7f4ccc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.237+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 shutdown_connections 2023-12-08T10:11:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:12.237+0000 7f4cd08be700 1 -- 172.21.15.170:0/1013765794 wait complete. 2023-12-08T10:11:12.240 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:13.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:12.236449+0000 mon.smithi023 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.170:0/1013765794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:13.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:13.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:12.350042+0000 mgr.smithi023.tgiigg (mgr.14164) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:14.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 -- 172.21.15.170:0/3233479245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 msgr2=0x7fb3a40f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:14.765 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 --2- 172.21.15.170:0/3233479245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f8fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fb398009fb0 tx=0x7fb398009b30).stop 2023-12-08T10:11:14.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 -- 172.21.15.170:0/3233479245 shutdown_connections 2023-12-08T10:11:14.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 --2- 172.21.15.170:0/3233479245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f8fc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:14.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 -- 172.21.15.170:0/3233479245 >> 172.21.15.170:0/3233479245 conn(0x7fb3a40f09a0 msgr2=0x7fb3a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:14.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 -- 172.21.15.170:0/3233479245 shutdown_connections 2023-12-08T10:11:14.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.760+0000 7fb3a9864700 1 -- 172.21.15.170:0/3233479245 wait complete. 2023-12-08T10:11:14.766 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a9864700 1 Processor -- start 2023-12-08T10:11:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a9864700 1 -- start start 2023-12-08T10:11:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a9864700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a9864700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40f4ec0 con 0x7fb3a4060140 2023-12-08T10:11:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a8862700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a8862700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:49328/0 (socket says 172.21.15.170:49328) 2023-12-08T10:11:14.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a8862700 1 -- 172.21.15.170:0/3140398088 learned_addr learned my addr 172.21.15.170:0/3140398088 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a8862700 1 -- 172.21.15.170:0/3140398088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb398005040 con 0x7fb3a4060140 2023-12-08T10:11:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.761+0000 7fb3a8862700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f6a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fb398013510 tx=0x7fb39800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.762+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb398009320 con 0x7fb3a4060140 2023-12-08T10:11:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.762+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a40f5060 con 0x7fb3a4060140 2023-12-08T10:11:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.762+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb398013f00 con 0x7fb3a4060140 2023-12-08T10:11:14.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.762+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb398014d90 con 0x7fb3a4060140 2023-12-08T10:11:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.762+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a40f54f0 con 0x7fb3a4060140 2023-12-08T10:11:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.763+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb398014420 con 0x7fb3a4060140 2023-12-08T10:11:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.763+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4059100 con 0x7fb3a4060140 2023-12-08T10:11:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.763+0000 7fb3a17fa700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb39402ff60 0x7fb394032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.763+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb398041000 con 0x7fb3a4060140 2023-12-08T10:11:14.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.763+0000 7fb3a3fff700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb39402ff60 0x7fb394032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:14.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.764+0000 7fb3a3fff700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb39402ff60 0x7fb394032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fb3900096a0 tx=0x7fb390006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:14.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.768+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3980095f0 con 0x7fb3a4060140 2023-12-08T10:11:14.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.973+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3a40fbce0 con 0x7fb3a4060140 2023-12-08T10:11:14.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.973+0000 7fb3a17fa700 1 -- 172.21.15.170:0/3140398088 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb39801db70 con 0x7fb3a4060140 2023-12-08T10:11:14.978 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:14.978 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb39402ff60 msgr2=0x7fb394032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb39402ff60 0x7fb394032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fb3900096a0 tx=0x7fb390006b40).stop 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 msgr2=0x7fb3a40f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f6a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fb398013510 tx=0x7fb39800b040).stop 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 shutdown_connections 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb39402ff60 0x7fb394032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 --2- 172.21.15.170:0/3140398088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a4060140 0x7fb3a40f6a40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 >> 172.21.15.170:0/3140398088 conn(0x7fb3a40f09a0 msgr2=0x7fb3a40f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 shutdown_connections 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:14.975+0000 7fb3a9864700 1 -- 172.21.15.170:0/3140398088 wait complete. 2023-12-08T10:11:14.979 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:14.350373+0000 mgr.smithi023.tgiigg (mgr.14164) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:14.974229+0000 mon.smithi023 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.170:0/3140398088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:16.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:16.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 -- 172.21.15.170:0/1754727355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 msgr2=0x7f14e00f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 --2- 172.21.15.170:0/1754727355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e00f8fe0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f14d0009fb0 tx=0x7f14d0009b30).stop 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 -- 172.21.15.170:0/1754727355 shutdown_connections 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 --2- 172.21.15.170:0/1754727355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e00f8fe0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 -- 172.21.15.170:0/1754727355 >> 172.21.15.170:0/1754727355 conn(0x7f14e00f09a0 msgr2=0x7f14e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 -- 172.21.15.170:0/1754727355 shutdown_connections 2023-12-08T10:11:17.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.429+0000 7f14e6ff1700 1 -- 172.21.15.170:0/1754727355 wait complete. 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.430+0000 7f14e6ff1700 1 Processor -- start 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.430+0000 7f14e6ff1700 1 -- start start 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.430+0000 7f14e6ff1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e01064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.430+0000 7f14e6ff1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e01069c0 con 0x7f14e0060120 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.430+0000 7f14e5fef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e01064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.430+0000 7f14e5fef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e01064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:49350/0 (socket says 172.21.15.170:49350) 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14e5fef700 1 -- 172.21.15.170:0/273922086 learned_addr learned my addr 172.21.15.170:0/273922086 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14e5fef700 1 -- 172.21.15.170:0/273922086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d0005040 con 0x7f14e0060120 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14e5fef700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e01064d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f14d0013510 tx=0x7f14d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d0009320 con 0x7f14e0060120 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e0106bc0 con 0x7f14e0060120 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14d0013f00 con 0x7f14e0060120 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14d0014d90 con 0x7f14e0060120 2023-12-08T10:11:17.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.431+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e0107050 con 0x7f14e0060120 2023-12-08T10:11:17.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.432+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14d00095f0 con 0x7f14e0060120 2023-12-08T10:11:17.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.432+0000 7f14d67fc700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f14cc02ff60 0x7f14cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:17.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.432+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e0045ad0 con 0x7f14e0060120 2023-12-08T10:11:17.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.432+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14d00410c0 con 0x7f14e0060120 2023-12-08T10:11:17.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.432+0000 7f14e57ee700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f14cc02ff60 0x7f14cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:17.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.433+0000 7f14e57ee700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f14cc02ff60 0x7f14cc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f14dc0096a0 tx=0x7f14dc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:17.441 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.437+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14d0014420 con 0x7f14e0060120 2023-12-08T10:11:17.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.645+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14e0107f00 con 0x7f14e0060120 2023-12-08T10:11:17.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.645+0000 7f14d67fc700 1 -- 172.21.15.170:0/273922086 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f14d0010230 con 0x7f14e0060120 2023-12-08T10:11:17.647 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:17.647 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:17.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f14cc02ff60 msgr2=0x7f14cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:17.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f14cc02ff60 0x7f14cc032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f14dc0096a0 tx=0x7f14dc006b40).stop 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 msgr2=0x7f14e01064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e01064d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f14d0013510 tx=0x7f14d000b040).stop 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 shutdown_connections 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f14cc02ff60 0x7f14cc032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 --2- 172.21.15.170:0/273922086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14e0060120 0x7f14e01064d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 >> 172.21.15.170:0/273922086 conn(0x7f14e00f09a0 msgr2=0x7f14e00f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 shutdown_connections 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:17.647+0000 7f14e6ff1700 1 -- 172.21.15.170:0/273922086 wait complete. 2023-12-08T10:11:17.649 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:16.350712+0000 mgr.smithi023.tgiigg (mgr.14164) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:17.646306+0000 mon.smithi023 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.170:0/273922086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:19.047 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:19.048 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:20.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:18.351025+0000 mgr.smithi023.tgiigg (mgr.14164) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:20.243 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.241+0000 7fc03cbf8700 1 -- 172.21.15.170:0/2234042000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 msgr2=0x7fc0380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.241+0000 7fc03cbf8700 1 --2- 172.21.15.170:0/2234042000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc0380f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fc028009fb0 tx=0x7fc028009b30).stop 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 -- 172.21.15.170:0/2234042000 shutdown_connections 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 --2- 172.21.15.170:0/2234042000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc0380f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 -- 172.21.15.170:0/2234042000 >> 172.21.15.170:0/2234042000 conn(0x7fc0380f1c70 msgr2=0x7fc0380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 -- 172.21.15.170:0/2234042000 shutdown_connections 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 -- 172.21.15.170:0/2234042000 wait complete. 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 Processor -- start 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 -- start start 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc03806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03cbf8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc038069e20 con 0x7fc0380f6510 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc03806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc03806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:49368/0 (socket says 172.21.15.170:49368) 2023-12-08T10:11:20.244 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.242+0000 7fc03659c700 1 -- 172.21.15.170:0/3594038276 learned_addr learned my addr 172.21.15.170:0/3594038276 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc03659c700 1 -- 172.21.15.170:0/3594038276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc028005040 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc03659c700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc03806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fc028013510 tx=0x7fc02800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc028009320 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc028013f00 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc038069fc0 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc028014d90 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.243+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03806a450 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.244+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0280095f0 con 0x7fc0380f6510 2023-12-08T10:11:20.246 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.244+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc038059100 con 0x7fc0380f6510 2023-12-08T10:11:20.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.244+0000 7fc02effd700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc02402ff60 0x7fc024032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:20.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.244+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc028040fe0 con 0x7fc0380f6510 2023-12-08T10:11:20.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.244+0000 7fc035d9b700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc02402ff60 0x7fc024032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:20.247 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.245+0000 7fc035d9b700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc02402ff60 0x7fc024032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc0200096a0 tx=0x7fc020006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:20.250 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.249+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc02801d7f0 con 0x7fc0380f6510 2023-12-08T10:11:20.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.458+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0380fbf60 con 0x7fc0380f6510 2023-12-08T10:11:20.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.458+0000 7fc02effd700 1 -- 172.21.15.170:0/3594038276 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc028010230 con 0x7fc0380f6510 2023-12-08T10:11:20.460 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:20.460 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:20.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc02402ff60 msgr2=0x7fc024032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:20.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc02402ff60 0x7fc024032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc0200096a0 tx=0x7fc020006b40).stop 2023-12-08T10:11:20.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 msgr2=0x7fc03806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:20.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc03806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fc028013510 tx=0x7fc02800b040).stop 2023-12-08T10:11:20.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 shutdown_connections 2023-12-08T10:11:20.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc02402ff60 0x7fc024032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:20.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 --2- 172.21.15.170:0/3594038276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0380f6510 0x7fc03806b9a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:20.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 >> 172.21.15.170:0/3594038276 conn(0x7fc0380f1c70 msgr2=0x7fc0380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:20.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 shutdown_connections 2023-12-08T10:11:20.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:20.460+0000 7fc03cbf8700 1 -- 172.21.15.170:0/3594038276 wait complete. 2023-12-08T10:11:20.462 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:21.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:20.459043+0000 mon.smithi023 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.170:0/3594038276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:21.834 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:21.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:20.351301+0000 mgr.smithi023.tgiigg (mgr.14164) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 -- 172.21.15.170:0/3452674046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 msgr2=0x7f4fc40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 --2- 172.21.15.170:0/3452674046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc40f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f4fb4009fb0 tx=0x7f4fb4009b30).stop 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 -- 172.21.15.170:0/3452674046 shutdown_connections 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 --2- 172.21.15.170:0/3452674046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc40f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 -- 172.21.15.170:0/3452674046 >> 172.21.15.170:0/3452674046 conn(0x7f4fc40f0b70 msgr2=0x7f4fc40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 -- 172.21.15.170:0/3452674046 shutdown_connections 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.966+0000 7f4fcc23f700 1 -- 172.21.15.170:0/3452674046 wait complete. 2023-12-08T10:11:22.969 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fcc23f700 1 Processor -- start 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fcc23f700 1 -- start start 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fcc23f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc4106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fcc23f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc4106c00 con 0x7f4fc40f5410 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fc9fdb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc4106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fc9fdb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc4106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36036/0 (socket says 172.21.15.170:36036) 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fc9fdb700 1 -- 172.21.15.170:0/1397153784 learned_addr learned my addr 172.21.15.170:0/1397153784 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fc9fdb700 1 -- 172.21.15.170:0/1397153784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb4005040 con 0x7f4fc40f5410 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.967+0000 7f4fc9fdb700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc4106710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f4fb4014000 tx=0x7f4fb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fb40095f0 con 0x7f4fc40f5410 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc4106e00 con 0x7f4fc40f5410 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fb4007e70 con 0x7f4fc40f5410 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fb4014ea0 con 0x7f4fc40f5410 2023-12-08T10:11:22.970 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc4107290 con 0x7f4fc40f5410 2023-12-08T10:11:22.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4fb4014690 con 0x7f4fc40f5410 2023-12-08T10:11:22.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.968+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc4045ad0 con 0x7f4fc40f5410 2023-12-08T10:11:22.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.969+0000 7f4fba7fc700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fb002ff60 0x7f4fb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:22.971 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.969+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4fb4040870 con 0x7f4fc40f5410 2023-12-08T10:11:22.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.969+0000 7f4fc97da700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fb002ff60 0x7f4fb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:22.972 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.969+0000 7f4fc97da700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fb002ff60 0x7f4fb0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4fc00096a0 tx=0x7f4fc0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:22.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:22.974+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4fb400f300 con 0x7f4fc40f5410 2023-12-08T10:11:23.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.179+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fc40fae60 con 0x7f4fc40f5410 2023-12-08T10:11:23.180 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.179+0000 7f4fba7fc700 1 -- 172.21.15.170:0/1397153784 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4fb400f300 con 0x7f4fc40f5410 2023-12-08T10:11:23.181 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:23.181 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:22.453565+0000 mon.smithi023 (mon.0) 223 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:11:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:22.455631+0000 mon.smithi023 (mon.0) 224 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fb002ff60 msgr2=0x7f4fb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fb002ff60 0x7f4fb0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f4fc00096a0 tx=0x7f4fc0006b40).stop 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 msgr2=0x7f4fc4106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc4106710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f4fb4014000 tx=0x7f4fb400b040).stop 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 shutdown_connections 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fb002ff60 0x7f4fb0032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 --2- 172.21.15.170:0/1397153784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40f5410 0x7f4fc4106710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 >> 172.21.15.170:0/1397153784 conn(0x7f4fc40f0b70 msgr2=0x7f4fc405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 shutdown_connections 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:23.181+0000 7f4fcc23f700 1 -- 172.21.15.170:0/1397153784 wait complete. 2023-12-08T10:11:23.183 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:22.351623+0000 mgr.smithi023.tgiigg (mgr.14164) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:23.180135+0000 mon.smithi023 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.170:0/1397153784' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:24.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:24.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:25.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 -- 172.21.15.170:0/2642171841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 msgr2=0x7f72640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:25.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 --2- 172.21.15.170:0/2642171841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72640f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f7254009fb0 tx=0x7f7254009b30).stop 2023-12-08T10:11:25.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 -- 172.21.15.170:0/2642171841 shutdown_connections 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 --2- 172.21.15.170:0/2642171841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72640f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 -- 172.21.15.170:0/2642171841 >> 172.21.15.170:0/2642171841 conn(0x7f72640f09a0 msgr2=0x7f72640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 -- 172.21.15.170:0/2642171841 shutdown_connections 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.627+0000 7f7269778700 1 -- 172.21.15.170:0/2642171841 wait complete. 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7269778700 1 Processor -- start 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7269778700 1 -- start start 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7269778700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72641091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7269778700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264060220 con 0x7f72640f51a0 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7263fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72641091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7263fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72641091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36062/0 (socket says 172.21.15.170:36062) 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.628+0000 7f7263fff700 1 -- 172.21.15.170:0/2032879187 learned_addr learned my addr 172.21.15.170:0/2032879187 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7263fff700 1 -- 172.21.15.170:0/2032879187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7254005040 con 0x7f72640f51a0 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7263fff700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72641091d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f7254009830 tx=0x7f725400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:25.631 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7254009320 con 0x7f72640f51a0 2023-12-08T10:11:25.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72540136c0 con 0x7f72640f51a0 2023-12-08T10:11:25.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7264060420 con 0x7f72640f51a0 2023-12-08T10:11:25.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f725401ee90 con 0x7f72640f51a0 2023-12-08T10:11:25.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.629+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7264109880 con 0x7f72640f51a0 2023-12-08T10:11:25.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.630+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f725401e420 con 0x7f72640f51a0 2023-12-08T10:11:25.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.630+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7264059100 con 0x7f72640f51a0 2023-12-08T10:11:25.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.630+0000 7f7260ff9700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f724c02ff10 0x7f724c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:25.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.630+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7254040f60 con 0x7f72640f51a0 2023-12-08T10:11:25.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.630+0000 7f72637fe700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f724c02ff10 0x7f724c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:25.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.630+0000 7f72637fe700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f724c02ff10 0x7f724c0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f72580096a0 tx=0x7f7258006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:25.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.634+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f725401c790 con 0x7f72640f51a0 2023-12-08T10:11:25.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.842+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72640fabe0 con 0x7f72640f51a0 2023-12-08T10:11:25.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.843+0000 7f7260ff9700 1 -- 172.21.15.170:0/2032879187 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f725402c2d0 con 0x7f72640f51a0 2023-12-08T10:11:25.844 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:25.844 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:25.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f724c02ff10 msgr2=0x7f724c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f724c02ff10 0x7f724c0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f72580096a0 tx=0x7f7258006b40).stop 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 msgr2=0x7f72641091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72641091d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f7254009830 tx=0x7f725400b040).stop 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 shutdown_connections 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f724c02ff10 0x7f724c0323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 --2- 172.21.15.170:0/2032879187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72640f51a0 0x7f72641091d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 >> 172.21.15.170:0/2032879187 conn(0x7f72640f09a0 msgr2=0x7f72640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 shutdown_connections 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:25.844+0000 7f7269778700 1 -- 172.21.15.170:0/2032879187 wait complete. 2023-12-08T10:11:25.846 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:26.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:24.352027+0000 mgr.smithi023.tgiigg (mgr.14164) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:26.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:25.843388+0000 mon.smithi023 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.170:0/2032879187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:27.230 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:27.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:28.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:26.352338+0000 mgr.smithi023.tgiigg (mgr.14164) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2038649112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 msgr2=0x7f7f480f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 --2- 172.21.15.170:0/2038649112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f480f8f80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f7f38009fb0 tx=0x7f7f38009b30).stop 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2038649112 shutdown_connections 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 --2- 172.21.15.170:0/2038649112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f480f8f80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2038649112 >> 172.21.15.170:0/2038649112 conn(0x7f7f480f09a0 msgr2=0x7f7f480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2038649112 shutdown_connections 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2038649112 wait complete. 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 Processor -- start 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- start start 2023-12-08T10:11:28.385 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f48106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.382+0000 7f7f4ef64700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f481069f0 con 0x7f7f48060180 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f4df62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f48106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f4df62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f48106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36078/0 (socket says 172.21.15.170:36078) 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f4df62700 1 -- 172.21.15.170:0/2135179937 learned_addr learned my addr 172.21.15.170:0/2135179937 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f4df62700 1 -- 172.21.15.170:0/2135179937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f38005040 con 0x7f7f48060180 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f4df62700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f48106500 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7f38013510 tx=0x7f7f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f38009320 con 0x7f7f48060180 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f48106bf0 con 0x7f7f48060180 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f38013f00 con 0x7f7f48060180 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.383+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f38014d90 con 0x7f7f48060180 2023-12-08T10:11:28.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.384+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f48107080 con 0x7f7f48060180 2023-12-08T10:11:28.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.384+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f380263a0 con 0x7f7f48060180 2023-12-08T10:11:28.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.384+0000 7f7f3e7fc700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7f3402ff60 0x7f7f34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:28.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.384+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f38040fe0 con 0x7f7f48060180 2023-12-08T10:11:28.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.385+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f48059100 con 0x7f7f48060180 2023-12-08T10:11:28.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.385+0000 7f7f4d761700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7f3402ff60 0x7f7f34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:28.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.385+0000 7f7f4d761700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7f3402ff60 0x7f7f34032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7f440096a0 tx=0x7f7f44006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:28.390 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.389+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f38012920 con 0x7f7f48060180 2023-12-08T10:11:28.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.595+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f48045ad0 con 0x7f7f48060180 2023-12-08T10:11:28.597 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.595+0000 7f7f3e7fc700 1 -- 172.21.15.170:0/2135179937 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f3802b020 con 0x7f7f48060180 2023-12-08T10:11:28.597 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:28.597 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:28.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7f3402ff60 msgr2=0x7f7f34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:28.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7f3402ff60 0x7f7f34032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7f440096a0 tx=0x7f7f44006b40).stop 2023-12-08T10:11:28.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 msgr2=0x7f7f48106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f48106500 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7f38013510 tx=0x7f7f3800b040).stop 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 shutdown_connections 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7f3402ff60 0x7f7f34032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 --2- 172.21.15.170:0/2135179937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f48060180 0x7f7f48106500 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 >> 172.21.15.170:0/2135179937 conn(0x7f7f480f09a0 msgr2=0x7f7f480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 shutdown_connections 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:28.597+0000 7f7f4ef64700 1 -- 172.21.15.170:0/2135179937 wait complete. 2023-12-08T10:11:28.599 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:29.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:28 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:28.596256+0000 mon.smithi023 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.170:0/2135179937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:29.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:29.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:28.352614+0000 mgr.smithi023.tgiigg (mgr.14164) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:31.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/230161590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 msgr2=0x7f6e680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:31.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 --2- 172.21.15.170:0/230161590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e680f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f6e58009fb0 tx=0x7f6e58009b30).stop 2023-12-08T10:11:31.126 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/230161590 shutdown_connections 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 --2- 172.21.15.170:0/230161590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e680f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/230161590 >> 172.21.15.170:0/230161590 conn(0x7f6e680f0b70 msgr2=0x7f6e680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/230161590 shutdown_connections 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.123+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/230161590 wait complete. 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6ccd1700 1 Processor -- start 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6ccd1700 1 -- start start 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6ccd1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e68108940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6ccd1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e68108e30 con 0x7f6e680f5410 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e68108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e68108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36096/0 (socket says 172.21.15.170:36096) 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6659c700 1 -- 172.21.15.170:0/3989295546 learned_addr learned my addr 172.21.15.170:0/3989295546 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.124+0000 7f6e6659c700 1 -- 172.21.15.170:0/3989295546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58005040 con 0x7f6e680f5410 2023-12-08T10:11:31.127 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.125+0000 7f6e6659c700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e68108940 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6e58014000 tx=0x7f6e5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:31.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.125+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e580095f0 con 0x7f6e680f5410 2023-12-08T10:11:31.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.125+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e68109030 con 0x7f6e680f5410 2023-12-08T10:11:31.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.125+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e58007e70 con 0x7f6e680f5410 2023-12-08T10:11:31.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.125+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e58014ea0 con 0x7f6e680f5410 2023-12-08T10:11:31.128 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.125+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e681094c0 con 0x7f6e680f5410 2023-12-08T10:11:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.126+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e58014690 con 0x7f6e680f5410 2023-12-08T10:11:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.126+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e68045ad0 con 0x7f6e680f5410 2023-12-08T10:11:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.126+0000 7f6e5effd700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e5402ff60 0x7f6e54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.126+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e58040870 con 0x7f6e680f5410 2023-12-08T10:11:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.126+0000 7f6e65d9b700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e5402ff60 0x7f6e54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.126+0000 7f6e65d9b700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e5402ff60 0x7f6e54032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6e500096a0 tx=0x7f6e50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:31.132 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.131+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e5800f300 con 0x7f6e680f5410 2023-12-08T10:11:31.339 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.338+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e680fae60 con 0x7f6e680f5410 2023-12-08T10:11:31.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.338+0000 7f6e5effd700 1 -- 172.21.15.170:0/3989295546 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e5800f300 con 0x7f6e680f5410 2023-12-08T10:11:31.340 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:31.340 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e5402ff60 msgr2=0x7f6e54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e5402ff60 0x7f6e54032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6e500096a0 tx=0x7f6e50006b40).stop 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 msgr2=0x7f6e68108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e68108940 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6e58014000 tx=0x7f6e5800b040).stop 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 shutdown_connections 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e5402ff60 0x7f6e54032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 --2- 172.21.15.170:0/3989295546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e680f5410 0x7f6e68108940 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 >> 172.21.15.170:0/3989295546 conn(0x7f6e680f0b70 msgr2=0x7f6e680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:31.342 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 shutdown_connections 2023-12-08T10:11:31.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:31.340+0000 7f6e6ccd1700 1 -- 172.21.15.170:0/3989295546 wait complete. 2023-12-08T10:11:31.343 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:30.352961+0000 mgr.smithi023.tgiigg (mgr.14164) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:31.339294+0000 mon.smithi023 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.170:0/3989295546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:32.729 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:32.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:33.902 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 -- 172.21.15.170:0/385067848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 msgr2=0x7f1ee80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:33.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 --2- 172.21.15.170:0/385067848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee80f90f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f1edc009fb0 tx=0x7f1edc009b30).stop 2023-12-08T10:11:33.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 -- 172.21.15.170:0/385067848 shutdown_connections 2023-12-08T10:11:33.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 --2- 172.21.15.170:0/385067848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee80f90f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:33.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 -- 172.21.15.170:0/385067848 >> 172.21.15.170:0/385067848 conn(0x7f1ee80f0b70 msgr2=0x7f1ee80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:33.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 -- 172.21.15.170:0/385067848 shutdown_connections 2023-12-08T10:11:33.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.900+0000 7f1eed81d700 1 -- 172.21.15.170:0/385067848 wait complete. 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1eed81d700 1 Processor -- start 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1eed81d700 1 -- start start 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1eed81d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1eed81d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee8106bd0 con 0x7f1ee8060280 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1ee6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1ee6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41020/0 (socket says 172.21.15.170:41020) 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.901+0000 7f1ee6ffd700 1 -- 172.21.15.170:0/3434648322 learned_addr learned my addr 172.21.15.170:0/3434648322 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:33.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1ee6ffd700 1 -- 172.21.15.170:0/3434648322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc005040 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1ee6ffd700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee81066e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f1edc009830 tx=0x7f1edc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1edc009320 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee8106dd0 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1edc0136c0 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1edc01ee90 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.902+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee8107260 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.903+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1edc014440 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.903+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee8045ad0 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.903+0000 7f1ecf7fe700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1ed002ff10 0x7f1ed00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.903+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1edc040fa0 con 0x7f1ee8060280 2023-12-08T10:11:33.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.903+0000 7f1ee67fc700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1ed002ff10 0x7f1ed00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:33.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.904+0000 7f1ee67fc700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1ed002ff10 0x7f1ed00323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f1ed80096a0 tx=0x7f1ed8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:33.910 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:33.908+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1edc0095f0 con 0x7f1ee8060280 2023-12-08T10:11:34.114 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.112+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ee81080b0 con 0x7f1ee8060280 2023-12-08T10:11:34.115 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.113+0000 7f1ecf7fe700 1 -- 172.21.15.170:0/3434648322 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1edc02c2d0 con 0x7f1ee8060280 2023-12-08T10:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:34.115 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1ed002ff10 msgr2=0x7f1ed00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1ed002ff10 0x7f1ed00323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f1ed80096a0 tx=0x7f1ed8006b40).stop 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 msgr2=0x7f1ee81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee81066e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f1edc009830 tx=0x7f1edc00b040).stop 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 shutdown_connections 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1ed002ff10 0x7f1ed00323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 --2- 172.21.15.170:0/3434648322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee8060280 0x7f1ee81066e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 >> 172.21.15.170:0/3434648322 conn(0x7f1ee80f0b70 msgr2=0x7f1ee80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 shutdown_connections 2023-12-08T10:11:34.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:34.114+0000 7f1eed81d700 1 -- 172.21.15.170:0/3434648322 wait complete. 2023-12-08T10:11:34.118 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:34.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:32.353348+0000 mgr.smithi023.tgiigg (mgr.14164) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:35.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:34 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:34.114311+0000 mon.smithi023 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.170:0/3434648322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:35.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:35.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:36.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:34.353754+0000 mgr.smithi023.tgiigg (mgr.14164) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:36.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.650+0000 7f1216c53700 1 -- 172.21.15.170:0/3174837839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 msgr2=0x7f12100fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:36.653 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.650+0000 7f1216c53700 1 --2- 172.21.15.170:0/3174837839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12100fb570 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f1204009fb0 tx=0x7f1204009b30).stop 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.650+0000 7f1216c53700 1 -- 172.21.15.170:0/3174837839 shutdown_connections 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.650+0000 7f1216c53700 1 --2- 172.21.15.170:0/3174837839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12100fb570 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.650+0000 7f1216c53700 1 -- 172.21.15.170:0/3174837839 >> 172.21.15.170:0/3174837839 conn(0x7f12100f0b70 msgr2=0x7f12100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.651+0000 7f1216c53700 1 -- 172.21.15.170:0/3174837839 shutdown_connections 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.651+0000 7f1216c53700 1 -- 172.21.15.170:0/3174837839 wait complete. 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.651+0000 7f1216c53700 1 Processor -- start 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.651+0000 7f1216c53700 1 -- start start 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.651+0000 7f1216c53700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12101088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:36.654 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.651+0000 7f1216c53700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1210108dc0 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f12149ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12101088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f12149ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12101088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41046/0 (socket says 172.21.15.170:41046) 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f12149ef700 1 -- 172.21.15.170:0/354104757 learned_addr learned my addr 172.21.15.170:0/354104757 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f12149ef700 1 -- 172.21.15.170:0/354104757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204005040 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f12149ef700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12101088d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f1204013510 tx=0x7f120400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1204009320 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1210108fc0 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1204013f00 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1210109450 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.652+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1204014dc0 con 0x7f1210060200 2023-12-08T10:11:36.656 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.653+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f120401d3d0 con 0x7f1210060200 2023-12-08T10:11:36.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.653+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1210045ad0 con 0x7f1210060200 2023-12-08T10:11:36.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.653+0000 7f120d7fa700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f120002ff60 0x7f1200032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:36.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.653+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12040413e0 con 0x7f1210060200 2023-12-08T10:11:36.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.654+0000 7f120ffff700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f120002ff60 0x7f1200032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:36.657 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.654+0000 7f120ffff700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f120002ff60 0x7f1200032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f11fc0096a0 tx=0x7f11fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:36.660 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.658+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f120401ccf0 con 0x7f1210060200 2023-12-08T10:11:36.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.864+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f121010a230 con 0x7f1210060200 2023-12-08T10:11:36.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.865+0000 7f120d7fa700 1 -- 172.21.15.170:0/354104757 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f120401cb10 con 0x7f1210060200 2023-12-08T10:11:36.868 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:36.868 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:36.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f120002ff60 msgr2=0x7f1200032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f120002ff60 0x7f1200032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f11fc0096a0 tx=0x7f11fc006b40).stop 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 msgr2=0x7f12101088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12101088d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f1204013510 tx=0x7f120400b040).stop 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 shutdown_connections 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f120002ff60 0x7f1200032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 --2- 172.21.15.170:0/354104757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1210060200 0x7f12101088d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 >> 172.21.15.170:0/354104757 conn(0x7f12100f0b70 msgr2=0x7f12100f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 shutdown_connections 2023-12-08T10:11:36.870 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:36.867+0000 7f1216c53700 1 -- 172.21.15.170:0/354104757 wait complete. 2023-12-08T10:11:36.871 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:37.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:36 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:36.866758+0000 mon.smithi023 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.170:0/354104757' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:38.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:38.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:38.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:36.354099+0000 mgr.smithi023.tgiigg (mgr.14164) 158 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:39.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- 172.21.15.170:0/921115520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 msgr2=0x7f93180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 --2- 172.21.15.170:0/921115520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93180f55c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9308009fb0 tx=0x7f9308009b30).stop 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- 172.21.15.170:0/921115520 shutdown_connections 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 --2- 172.21.15.170:0/921115520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93180f55c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- 172.21.15.170:0/921115520 >> 172.21.15.170:0/921115520 conn(0x7f93180f09a0 msgr2=0x7f93180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- 172.21.15.170:0/921115520 shutdown_connections 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- 172.21.15.170:0/921115520 wait complete. 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 Processor -- start 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- start start 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93181064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:39.367 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.364+0000 7f931e14e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93181069d0 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f931d14c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93181064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f931d14c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93181064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41068/0 (socket says 172.21.15.170:41068) 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f931d14c700 1 -- 172.21.15.170:0/1626149631 learned_addr learned my addr 172.21.15.170:0/1626149631 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f931d14c700 1 -- 172.21.15.170:0/1626149631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9308005040 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f931d14c700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93181064e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9308013510 tx=0x7f930800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9308009320 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9318106bd0 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9308013f00 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.365+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9308014dc0 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.366+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9318107060 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.366+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93080095f0 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.366+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9318045ad0 con 0x7f93180f51a0 2023-12-08T10:11:39.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.366+0000 7f930dffb700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:39.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.366+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93080410a0 con 0x7f93180f51a0 2023-12-08T10:11:39.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.367+0000 7f931c94b700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:39.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.367+0000 7f931c94b700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f930402ff60 0x7f9304032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f93140096a0 tx=0x7f9314006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:39.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.371+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9308014420 con 0x7f93180f51a0 2023-12-08T10:11:39.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.580+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9318107eb0 con 0x7f93180f51a0 2023-12-08T10:11:39.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.581+0000 7f930dffb700 1 -- 172.21.15.170:0/1626149631 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f930801c920 con 0x7f93180f51a0 2023-12-08T10:11:39.583 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:39.583 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f930402ff60 msgr2=0x7f9304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f930402ff60 0x7f9304032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f93140096a0 tx=0x7f9314006b40).stop 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 msgr2=0x7f93181064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93181064e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9308013510 tx=0x7f930800b040).stop 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 shutdown_connections 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f930402ff60 0x7f9304032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 --2- 172.21.15.170:0/1626149631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93180f51a0 0x7f93181064e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 >> 172.21.15.170:0/1626149631 conn(0x7f93180f09a0 msgr2=0x7f931805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 shutdown_connections 2023-12-08T10:11:39.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:39.582+0000 7f931e14e700 1 -- 172.21.15.170:0/1626149631 wait complete. 2023-12-08T10:11:39.586 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:40.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:38.354455+0000 mgr.smithi023.tgiigg (mgr.14164) 159 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:40.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:39.582381+0000 mon.smithi023 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.170:0/1626149631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:40.964 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:40.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:42.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 -- 172.21.15.170:0/1189345590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 msgr2=0x7fa44c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:42.074 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 --2- 172.21.15.170:0/1189345590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c0f55c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fa440009fb0 tx=0x7fa440009b30).stop 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 -- 172.21.15.170:0/1189345590 shutdown_connections 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 --2- 172.21.15.170:0/1189345590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c0f55c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 -- 172.21.15.170:0/1189345590 >> 172.21.15.170:0/1189345590 conn(0x7fa44c0f09a0 msgr2=0x7fa44c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 -- 172.21.15.170:0/1189345590 shutdown_connections 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.071+0000 7fa451b3e700 1 -- 172.21.15.170:0/1189345590 wait complete. 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.072+0000 7fa451b3e700 1 Processor -- start 2023-12-08T10:11:42.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.072+0000 7fa451b3e700 1 -- start start 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.072+0000 7fa451b3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.072+0000 7fa451b3e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44c1058a0 con 0x7fa44c0f51a0 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa450b3c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa450b3c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:41096/0 (socket says 172.21.15.170:41096) 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa450b3c700 1 -- 172.21.15.170:0/3052719617 learned_addr learned my addr 172.21.15.170:0/3052719617 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa450b3c700 1 -- 172.21.15.170:0/3052719617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa440005040 con 0x7fa44c0f51a0 2023-12-08T10:11:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa450b3c700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c1053b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fa440013510 tx=0x7fa44000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:42.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa440009320 con 0x7fa44c0f51a0 2023-12-08T10:11:42.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c105aa0 con 0x7fa44c0f51a0 2023-12-08T10:11:42.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa440013f00 con 0x7fa44c0f51a0 2023-12-08T10:11:42.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa440014dc0 con 0x7fa44c0f51a0 2023-12-08T10:11:42.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.073+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44c105f30 con 0x7fa44c0f51a0 2023-12-08T10:11:42.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.074+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4400095f0 con 0x7fa44c0f51a0 2023-12-08T10:11:42.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.074+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44c045ad0 con 0x7fa44c0f51a0 2023-12-08T10:11:42.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.074+0000 7fa4497fa700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa43c030000 0x7fa43c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:42.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.074+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa440041050 con 0x7fa44c0f51a0 2023-12-08T10:11:42.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.075+0000 7fa44bfff700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa43c030000 0x7fa43c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:42.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.075+0000 7fa44bfff700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa43c030000 0x7fa43c0324c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa4380096a0 tx=0x7fa438006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:42.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.079+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa44006d2b0 con 0x7fa44c0f51a0 2023-12-08T10:11:42.288 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.286+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa44c106d80 con 0x7fa44c0f51a0 2023-12-08T10:11:42.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.287+0000 7fa4497fa700 1 -- 172.21.15.170:0/3052719617 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa440014420 con 0x7fa44c0f51a0 2023-12-08T10:11:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:42.289 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa43c030000 msgr2=0x7fa43c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa43c030000 0x7fa43c0324c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fa4380096a0 tx=0x7fa438006b40).stop 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 msgr2=0x7fa44c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c1053b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fa440013510 tx=0x7fa44000b040).stop 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 shutdown_connections 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa43c030000 0x7fa43c0324c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 --2- 172.21.15.170:0/3052719617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa44c0f51a0 0x7fa44c1053b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 >> 172.21.15.170:0/3052719617 conn(0x7fa44c0f09a0 msgr2=0x7fa44c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.288+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 shutdown_connections 2023-12-08T10:11:42.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:42.289+0000 7fa451b3e700 1 -- 172.21.15.170:0/3052719617 wait complete. 2023-12-08T10:11:42.292 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:42.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:40.354813+0000 mgr.smithi023.tgiigg (mgr.14164) 160 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:43.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:42 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:42.288535+0000 mon.smithi023 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.170:0/3052719617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:43.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:43.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:44.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:42.355145+0000 mgr.smithi023.tgiigg (mgr.14164) 161 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:44.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.770+0000 7f462fa59700 1 -- 172.21.15.170:0/4152645614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 msgr2=0x7f46280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.770+0000 7f462fa59700 1 --2- 172.21.15.170:0/4152645614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f46280f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4618009fb0 tx=0x7f4618009b30).stop 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.770+0000 7f462fa59700 1 -- 172.21.15.170:0/4152645614 shutdown_connections 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.770+0000 7f462fa59700 1 --2- 172.21.15.170:0/4152645614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f46280f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.770+0000 7f462fa59700 1 -- 172.21.15.170:0/4152645614 >> 172.21.15.170:0/4152645614 conn(0x7f46280f1c70 msgr2=0x7f46280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462fa59700 1 -- 172.21.15.170:0/4152645614 shutdown_connections 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462fa59700 1 -- 172.21.15.170:0/4152645614 wait complete. 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462fa59700 1 Processor -- start 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462fa59700 1 -- start start 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462fa59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f462806b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462fa59700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4628069e20 con 0x7f46280f6510 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462d7f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f462806b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:44.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.771+0000 7f462d7f5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f462806b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48782/0 (socket says 172.21.15.170:48782) 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f462d7f5700 1 -- 172.21.15.170:0/2326624600 learned_addr learned my addr 172.21.15.170:0/2326624600 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f462d7f5700 1 -- 172.21.15.170:0/2326624600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618005040 con 0x7f46280f6510 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f462d7f5700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f462806b930 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4618013510 tx=0x7f461800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4618009320 con 0x7f46280f6510 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462806a020 con 0x7f46280f6510 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4618013f00 con 0x7f46280f6510 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4618014d90 con 0x7f46280f6510 2023-12-08T10:11:44.776 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.772+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462806a4b0 con 0x7f46280f6510 2023-12-08T10:11:44.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.773+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46180263a0 con 0x7f46280f6510 2023-12-08T10:11:44.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.773+0000 7f461dffb700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f461402ff60 0x7f4614032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:44.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.773+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4618041070 con 0x7f46280f6510 2023-12-08T10:11:44.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.773+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4628059100 con 0x7f46280f6510 2023-12-08T10:11:44.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.774+0000 7f462cff4700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f461402ff60 0x7f4614032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:44.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.774+0000 7f462cff4700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f461402ff60 0x7f4614032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f46240096a0 tx=0x7f4624006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:44.780 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.778+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4618012920 con 0x7f46280f6510 2023-12-08T10:11:44.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.982+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46280fbf60 con 0x7f46280f6510 2023-12-08T10:11:44.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.983+0000 7f461dffb700 1 -- 172.21.15.170:0/2326624600 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f461800f340 con 0x7f46280f6510 2023-12-08T10:11:44.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:44.986 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:44.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f461402ff60 msgr2=0x7f4614032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:44.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f461402ff60 0x7f4614032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f46240096a0 tx=0x7f4624006b40).stop 2023-12-08T10:11:44.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 msgr2=0x7f462806b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f462806b930 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4618013510 tx=0x7f461800b040).stop 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 shutdown_connections 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f461402ff60 0x7f4614032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 --2- 172.21.15.170:0/2326624600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46280f6510 0x7f462806b930 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 >> 172.21.15.170:0/2326624600 conn(0x7f46280f1c70 msgr2=0x7f46280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 shutdown_connections 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:44.985+0000 7f462fa59700 1 -- 172.21.15.170:0/2326624600 wait complete. 2023-12-08T10:11:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:46.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:46.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:46.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:44.355522+0000 mgr.smithi023.tgiigg (mgr.14164) 162 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:46.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:44.984869+0000 mon.smithi023 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.170:0/2326624600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:47.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 -- 172.21.15.170:0/179550404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 msgr2=0x7f92240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:47.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 --2- 172.21.15.170:0/179550404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92240f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f9218009fb0 tx=0x7f9218009b30).stop 2023-12-08T10:11:47.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 -- 172.21.15.170:0/179550404 shutdown_connections 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 --2- 172.21.15.170:0/179550404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92240f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 -- 172.21.15.170:0/179550404 >> 172.21.15.170:0/179550404 conn(0x7f92240f0b70 msgr2=0x7f92240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 -- 172.21.15.170:0/179550404 shutdown_connections 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.461+0000 7f922aeb1700 1 -- 172.21.15.170:0/179550404 wait complete. 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.462+0000 7f922aeb1700 1 Processor -- start 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.462+0000 7f922aeb1700 1 -- start start 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.462+0000 7f922aeb1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92241066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:47.465 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.462+0000 7f922aeb1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9224106bc0 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.462+0000 7f9228c4d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92241066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f9228c4d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92241066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48798/0 (socket says 172.21.15.170:48798) 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f9228c4d700 1 -- 172.21.15.170:0/2762080317 learned_addr learned my addr 172.21.15.170:0/2762080317 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f9228c4d700 1 -- 172.21.15.170:0/2762080317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9218005040 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f9228c4d700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92241066d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9218014000 tx=0x7f921800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92180095f0 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9224106dc0 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9218007e70 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9224107250 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.463+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9218014ea0 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.464+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9218025070 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.464+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9204004fa0 con 0x7f92240f5410 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.464+0000 7f92217fa700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f921402ffb0 0x7f9214032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:47.467 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.464+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9218040890 con 0x7f92240f5410 2023-12-08T10:11:47.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.465+0000 7f9223fff700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f921402ffb0 0x7f9214032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:47.468 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.465+0000 7f9223fff700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f921402ffb0 0x7f9214032470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f92100096a0 tx=0x7f9210006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:47.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.469+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f921801d4b0 con 0x7f92240f5410 2023-12-08T10:11:47.677 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.675+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9204005670 con 0x7f92240f5410 2023-12-08T10:11:47.678 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.676+0000 7f92217fa700 1 -- 172.21.15.170:0/2762080317 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f921800f300 con 0x7f92240f5410 2023-12-08T10:11:47.678 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:47.678 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.677+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f921402ffb0 msgr2=0x7f9214032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.677+0000 7f922aeb1700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f921402ffb0 0x7f9214032470 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f92100096a0 tx=0x7f9210006b40).stop 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.677+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 msgr2=0x7f92241066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.677+0000 7f922aeb1700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92241066d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f9218014000 tx=0x7f921800b040).stop 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.678+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 shutdown_connections 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.678+0000 7f922aeb1700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f921402ffb0 0x7f9214032470 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.678+0000 7f922aeb1700 1 --2- 172.21.15.170:0/2762080317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92240f5410 0x7f92241066d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.678+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 >> 172.21.15.170:0/2762080317 conn(0x7f92240f0b70 msgr2=0x7f922405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.678+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 shutdown_connections 2023-12-08T10:11:47.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:47.678+0000 7f922aeb1700 1 -- 172.21.15.170:0/2762080317 wait complete. 2023-12-08T10:11:47.681 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:48.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:46.355859+0000 mgr.smithi023.tgiigg (mgr.14164) 163 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:48.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:47.677485+0000 mon.smithi023 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.170:0/2762080317' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:49.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:49.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 -- 172.21.15.170:0/1463434722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 msgr2=0x7f27180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 --2- 172.21.15.170:0/1463434722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f27180f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f270c009fb0 tx=0x7f270c009b30).stop 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 -- 172.21.15.170:0/1463434722 shutdown_connections 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 --2- 172.21.15.170:0/1463434722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f27180f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 -- 172.21.15.170:0/1463434722 >> 172.21.15.170:0/1463434722 conn(0x7f27180f1c70 msgr2=0x7f27180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 -- 172.21.15.170:0/1463434722 shutdown_connections 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.256+0000 7f271dafb700 1 -- 172.21.15.170:0/1463434722 wait complete. 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.257+0000 7f271dafb700 1 Processor -- start 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.257+0000 7f271dafb700 1 -- start start 2023-12-08T10:11:50.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.257+0000 7f271dafb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f271806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:50.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.257+0000 7f271dafb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2718069e20 con 0x7f27180f6510 2023-12-08T10:11:50.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.257+0000 7f27177fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f271806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:50.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f27177fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f271806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48822/0 (socket says 172.21.15.170:48822) 2023-12-08T10:11:50.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f27177fe700 1 -- 172.21.15.170:0/2875419684 learned_addr learned my addr 172.21.15.170:0/2875419684 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f27177fe700 1 -- 172.21.15.170:0/2875419684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c005040 con 0x7f27180f6510 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f27177fe700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f271806b9a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f270c013510 tx=0x7f270c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f270c009320 con 0x7f27180f6510 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2718069fc0 con 0x7f27180f6510 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f270c013f00 con 0x7f27180f6510 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f270c014d90 con 0x7f27180f6510 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.258+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271806a450 con 0x7f27180f6510 2023-12-08T10:11:50.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.259+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f270c0263a0 con 0x7f27180f6510 2023-12-08T10:11:50.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.259+0000 7f26fffff700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f270002ffb0 0x7f2700032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:50.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.259+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f270c0410b0 con 0x7f27180f6510 2023-12-08T10:11:50.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.259+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2718059100 con 0x7f27180f6510 2023-12-08T10:11:50.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.259+0000 7f2716ffd700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f270002ffb0 0x7f2700032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:50.264 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.260+0000 7f2716ffd700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f270002ffb0 0x7f2700032470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f27080096a0 tx=0x7f2708006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:50.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.264+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f270c012920 con 0x7f27180f6510 2023-12-08T10:11:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:48.356294+0000 mgr.smithi023.tgiigg (mgr.14164) 164 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:50.474 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.472+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f271806ab50 con 0x7f27180f6510 2023-12-08T10:11:50.475 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.473+0000 7f26fffff700 1 -- 172.21.15.170:0/2875419684 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f270c02b020 con 0x7f27180f6510 2023-12-08T10:11:50.475 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:50.475 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f270002ffb0 msgr2=0x7f2700032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f270002ffb0 0x7f2700032470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f27080096a0 tx=0x7f2708006b40).stop 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 msgr2=0x7f271806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f271806b9a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f270c013510 tx=0x7f270c00b040).stop 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 shutdown_connections 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f270002ffb0 0x7f2700032470 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 --2- 172.21.15.170:0/2875419684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27180f6510 0x7f271806b9a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 >> 172.21.15.170:0/2875419684 conn(0x7f27180f1c70 msgr2=0x7f27180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 shutdown_connections 2023-12-08T10:11:50.477 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:50.474+0000 7f271dafb700 1 -- 172.21.15.170:0/2875419684 wait complete. 2023-12-08T10:11:50.478 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:51.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:50.474611+0000 mon.smithi023 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.170:0/2875419684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:51.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:51.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:52.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:50.356654+0000 mgr.smithi023.tgiigg (mgr.14164) 165 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:52.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1994920611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 msgr2=0x7f34b00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:52.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 --2- 172.21.15.170:0/1994920611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b00f90f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f34a4009fb0 tx=0x7f34a4009b30).stop 2023-12-08T10:11:52.994 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1994920611 shutdown_connections 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 --2- 172.21.15.170:0/1994920611 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b00f90f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1994920611 >> 172.21.15.170:0/1994920611 conn(0x7f34b00f0b70 msgr2=0x7f34b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1994920611 shutdown_connections 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.990+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1994920611 wait complete. 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b7bf9700 1 Processor -- start 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b7bf9700 1 -- start start 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b7bf9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b7bf9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b0106bd0 con 0x7f34b0060280 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b5995700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b5995700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54636/0 (socket says 172.21.15.170:54636) 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.991+0000 7f34b5995700 1 -- 172.21.15.170:0/1634990070 learned_addr learned my addr 172.21.15.170:0/1634990070 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34b5995700 1 -- 172.21.15.170:0/1634990070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a4005040 con 0x7f34b0060280 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34b5995700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b01066e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f34a4013510 tx=0x7f34a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:52.995 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a4009320 con 0x7f34b0060280 2023-12-08T10:11:52.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b0106dd0 con 0x7f34b0060280 2023-12-08T10:11:52.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34a4013f00 con 0x7f34b0060280 2023-12-08T10:11:52.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34a4014dc0 con 0x7f34b0060280 2023-12-08T10:11:52.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.992+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b0107260 con 0x7f34b0060280 2023-12-08T10:11:52.996 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.993+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34a40095f0 con 0x7f34b0060280 2023-12-08T10:11:52.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.993+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b0045ad0 con 0x7f34b0060280 2023-12-08T10:11:52.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.993+0000 7f34a27fc700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f349c02ffb0 0x7f349c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:52.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.993+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34a40410c0 con 0x7f34b0060280 2023-12-08T10:11:52.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.993+0000 7f34b5194700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f349c02ffb0 0x7f349c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:52.997 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.994+0000 7f34b5194700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f349c02ffb0 0x7f349c032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f34ac0096a0 tx=0x7f34ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:53.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:52.999+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34a4014420 con 0x7f34b0060280 2023-12-08T10:11:53.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.205+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34b0108110 con 0x7f34b0060280 2023-12-08T10:11:53.207 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.205+0000 7f34a27fc700 1 -- 172.21.15.170:0/1634990070 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f34a4010230 con 0x7f34b0060280 2023-12-08T10:11:53.208 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:53.208 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:53.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f349c02ffb0 msgr2=0x7f349c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:53.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f349c02ffb0 0x7f349c032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f34ac0096a0 tx=0x7f34ac006b40).stop 2023-12-08T10:11:53.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 msgr2=0x7f34b01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:53.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b01066e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f34a4013510 tx=0x7f34a400b040).stop 2023-12-08T10:11:53.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 shutdown_connections 2023-12-08T10:11:53.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f349c02ffb0 0x7f349c032470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:53.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 --2- 172.21.15.170:0/1634990070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34b0060280 0x7f34b01066e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:53.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 >> 172.21.15.170:0/1634990070 conn(0x7f34b00f0b70 msgr2=0x7f34b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:53.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 shutdown_connections 2023-12-08T10:11:53.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:53.207+0000 7f34b7bf9700 1 -- 172.21.15.170:0/1634990070 wait complete. 2023-12-08T10:11:53.210 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:52.356998+0000 mgr.smithi023.tgiigg (mgr.14164) 166 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:53.207150+0000 mon.smithi023 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.170:0/1634990070' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:54.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:54.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:55.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- 172.21.15.170:0/2363583396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 msgr2=0x7fc72c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 --2- 172.21.15.170:0/2363583396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c0f66c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fc724009960 tx=0x7fc724004d10).stop 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- 172.21.15.170:0/2363583396 shutdown_connections 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 --2- 172.21.15.170:0/2363583396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c0f66c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- 172.21.15.170:0/2363583396 >> 172.21.15.170:0/2363583396 conn(0x7fc72c0f1ac0 msgr2=0x7fc72c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- 172.21.15.170:0/2363583396 shutdown_connections 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- 172.21.15.170:0/2363583396 wait complete. 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 Processor -- start 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- start start 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.728+0000 7fc732792700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72c1069f0 con 0x7fc72c0f62a0 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc731790700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc731790700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54654/0 (socket says 172.21.15.170:54654) 2023-12-08T10:11:55.731 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc731790700 1 -- 172.21.15.170:0/435069114 learned_addr learned my addr 172.21.15.170:0/435069114 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:55.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc731790700 1 -- 172.21.15.170:0/435069114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc724009640 con 0x7fc72c0f62a0 2023-12-08T10:11:55.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc731790700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c106500 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc724007bd0 tx=0x7fc724004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:55.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc724018070 con 0x7fc72c0f62a0 2023-12-08T10:11:55.732 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.729+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72c106bf0 con 0x7fc72c0f62a0 2023-12-08T10:11:55.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.730+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72c107080 con 0x7fc72c0f62a0 2023-12-08T10:11:55.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.730+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc72400b870 con 0x7fc72c0f62a0 2023-12-08T10:11:55.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.730+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc72401c560 con 0x7fc72c0f62a0 2023-12-08T10:11:55.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.730+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc724024060 con 0x7fc72c0f62a0 2023-12-08T10:11:55.733 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.730+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c045ad0 con 0x7fc72c0f62a0 2023-12-08T10:11:55.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.731+0000 7fc729ffb700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc71402ff60 0x7fc714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:55.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.731+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc724040d20 con 0x7fc72c0f62a0 2023-12-08T10:11:55.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.731+0000 7fc730f8f700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc71402ff60 0x7fc714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:55.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.731+0000 7fc730f8f700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc71402ff60 0x7fc714032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc7200096a0 tx=0x7fc720006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:55.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.735+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc724011320 con 0x7fc72c0f62a0 2023-12-08T10:11:55.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.943+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc72c107e80 con 0x7fc72c0f62a0 2023-12-08T10:11:55.945 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.944+0000 7fc729ffb700 1 -- 172.21.15.170:0/435069114 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc724011320 con 0x7fc72c0f62a0 2023-12-08T10:11:55.946 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:55.946 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:55.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.945+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc71402ff60 msgr2=0x7fc714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:55.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.945+0000 7fc732792700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc71402ff60 0x7fc714032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc7200096a0 tx=0x7fc720006b40).stop 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.945+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 msgr2=0x7fc72c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.945+0000 7fc732792700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c106500 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc724007bd0 tx=0x7fc724004d10).stop 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.946+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 shutdown_connections 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.946+0000 7fc732792700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fc71402ff60 0x7fc714032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.946+0000 7fc732792700 1 --2- 172.21.15.170:0/435069114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc72c0f62a0 0x7fc72c106500 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.946+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 >> 172.21.15.170:0/435069114 conn(0x7fc72c0f1ac0 msgr2=0x7fc72c0f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.946+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 shutdown_connections 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:55.946+0000 7fc732792700 1 -- 172.21.15.170:0/435069114 wait complete. 2023-12-08T10:11:55.948 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:54.357364+0000 mgr.smithi023.tgiigg (mgr.14164) 167 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:55.944558+0000 mon.smithi023 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.170:0/435069114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:11:57.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:11:57.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:11:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:56.357751+0000 mgr.smithi023.tgiigg (mgr.14164) 168 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:11:58.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 -- 172.21.15.170:0/3921160089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 msgr2=0x7f4ffc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:58.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 --2- 172.21.15.170:0/3921160089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f90f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4ff8009fb0 tx=0x7f4ff8009b30).stop 2023-12-08T10:11:58.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 -- 172.21.15.170:0/3921160089 shutdown_connections 2023-12-08T10:11:58.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 --2- 172.21.15.170:0/3921160089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f90f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:58.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 -- 172.21.15.170:0/3921160089 >> 172.21.15.170:0/3921160089 conn(0x7f4ffc0f0b70 msgr2=0x7f4ffc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:58.458 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 -- 172.21.15.170:0/3921160089 shutdown_connections 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.455+0000 7f500364a700 1 -- 172.21.15.170:0/3921160089 wait complete. 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f500364a700 1 Processor -- start 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f500364a700 1 -- start start 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f500364a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f500364a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0f4ff0 con 0x7f4ffc060280 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f50013e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f50013e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54678/0 (socket says 172.21.15.170:54678) 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f50013e6700 1 -- 172.21.15.170:0/2395500023 learned_addr learned my addr 172.21.15.170:0/2395500023 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f50013e6700 1 -- 172.21.15.170:0/2395500023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff8005040 con 0x7f4ffc060280 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.456+0000 7f50013e6700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f6b70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4ff8013510 tx=0x7f4ff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.457+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8009320 con 0x7f4ffc060280 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.457+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc0f5190 con 0x7f4ffc060280 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.457+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ff8013f00 con 0x7f4ffc060280 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.457+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8014dc0 con 0x7f4ffc060280 2023-12-08T10:11:58.459 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.457+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc0f5620 con 0x7f4ffc060280 2023-12-08T10:11:58.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.458+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ff8014420 con 0x7f4ffc060280 2023-12-08T10:11:58.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.458+0000 7f4fedffb700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fe802ff60 0x7f4fe8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:11:58.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.458+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc059100 con 0x7f4ffc060280 2023-12-08T10:11:58.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.458+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ff80410a0 con 0x7f4ffc060280 2023-12-08T10:11:58.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.458+0000 7f5000be5700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fe802ff60 0x7f4fe8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:11:58.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.458+0000 7f5000be5700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fe802ff60 0x7f4fe8032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4ff00096a0 tx=0x7f4ff0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:11:58.464 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.463+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ff80095f0 con 0x7f4ffc060280 2023-12-08T10:11:58.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.667+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ffc0fbf50 con 0x7f4ffc060280 2023-12-08T10:11:58.669 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.668+0000 7f4fedffb700 1 -- 172.21.15.170:0/2395500023 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ff802b020 con 0x7f4ffc060280 2023-12-08T10:11:58.669 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:11:58.670 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fe802ff60 msgr2=0x7f4fe8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fe802ff60 0x7f4fe8032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4ff00096a0 tx=0x7f4ff0006b40).stop 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 msgr2=0x7f4ffc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f6b70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4ff8013510 tx=0x7f4ff800b040).stop 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 shutdown_connections 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4fe802ff60 0x7f4fe8032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:58.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 --2- 172.21.15.170:0/2395500023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ffc060280 0x7f4ffc0f6b70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:11:58.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 >> 172.21.15.170:0/2395500023 conn(0x7f4ffc0f0b70 msgr2=0x7f4ffc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:11:58.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 shutdown_connections 2023-12-08T10:11:58.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:11:58.670+0000 7f500364a700 1 -- 172.21.15.170:0/2395500023 wait complete. 2023-12-08T10:11:58.672 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:11:59.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:58 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:11:58.668853+0000 mon.smithi023 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.170:0/2395500023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:00.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:00.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:11:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:11:58.358105+0000 mgr.smithi023.tgiigg (mgr.14164) 169 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:01.154 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 -- 172.21.15.170:0/745509253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 msgr2=0x7f822c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 --2- 172.21.15.170:0/745509253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0f55c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f821c009fb0 tx=0x7f821c009b30).stop 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 -- 172.21.15.170:0/745509253 shutdown_connections 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 --2- 172.21.15.170:0/745509253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0f55c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 -- 172.21.15.170:0/745509253 >> 172.21.15.170:0/745509253 conn(0x7f822c0f09a0 msgr2=0x7f822c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 -- 172.21.15.170:0/745509253 shutdown_connections 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 -- 172.21.15.170:0/745509253 wait complete. 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 Processor -- start 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.152+0000 7f823157c700 1 -- start start 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f823157c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0fb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f823157c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c0f9830 con 0x7f822c0f51a0 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f822bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0fb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f822bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0fb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:54686/0 (socket says 172.21.15.170:54686) 2023-12-08T10:12:01.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f822bfff700 1 -- 172.21.15.170:0/3550333126 learned_addr learned my addr 172.21.15.170:0/3550333126 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f822bfff700 1 -- 172.21.15.170:0/3550333126 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c005040 con 0x7f822c0f51a0 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f822bfff700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0fb3b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f821c013510 tx=0x7f821c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821c009320 con 0x7f822c0f51a0 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f821c013f00 con 0x7f822c0f51a0 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c0f99d0 con 0x7f822c0f51a0 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821c014dc0 con 0x7f822c0f51a0 2023-12-08T10:12:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.153+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c0f9e60 con 0x7f822c0f51a0 2023-12-08T10:12:01.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.154+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f821c01d3d0 con 0x7f822c0f51a0 2023-12-08T10:12:01.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.154+0000 7f8228ff9700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f821402ff60 0x7f8214032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:01.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.154+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822c059100 con 0x7f822c0f51a0 2023-12-08T10:12:01.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.154+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f821c0413b0 con 0x7f822c0f51a0 2023-12-08T10:12:01.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.154+0000 7f822b7fe700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f821402ff60 0x7f8214032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:01.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.155+0000 7f822b7fe700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f821402ff60 0x7f8214032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f82200096a0 tx=0x7f8220006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:01.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.159+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f821c01cb10 con 0x7f822c0f51a0 2023-12-08T10:12:01.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:00.169793+0000 mon.smithi023 (mon.0) 239 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi023", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-08T10:12:01.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:00.172516+0000 mon.smithi023 (mon.0) 240 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi023", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-08T10:12:01.176 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:00.173322+0000 mon.smithi023 (mon.0) 241 : audit [DBG] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-08T10:12:01.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.365+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f822c0fbce0 con 0x7f822c0f51a0 2023-12-08T10:12:01.366 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.365+0000 7f8228ff9700 1 -- 172.21.15.170:0/3550333126 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f821c01cb10 con 0x7f822c0f51a0 2023-12-08T10:12:01.367 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:01.367 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:01.368 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f821402ff60 msgr2=0x7f8214032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:01.368 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f821402ff60 0x7f8214032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f82200096a0 tx=0x7f8220006b40).stop 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 msgr2=0x7f822c0fb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0fb3b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f821c013510 tx=0x7f821c00b040).stop 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 shutdown_connections 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f821402ff60 0x7f8214032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 --2- 172.21.15.170:0/3550333126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f822c0f51a0 0x7f822c0fb3b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 >> 172.21.15.170:0/3550333126 conn(0x7f822c0f09a0 msgr2=0x7f822c0f1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 shutdown_connections 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:01.367+0000 7f823157c700 1 -- 172.21.15.170:0/3550333126 wait complete. 2023-12-08T10:12:01.369 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:02.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:12:00.168561+0000 mgr.smithi023.tgiigg (mgr.14164) 170 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi023 ... 2023-12-08T10:12:02.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-08T10:12:02.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-08T10:12:02.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:07:37Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:08:15Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:08:53Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:10:07Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:10:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:11:22Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Traceback (most recent call last): 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: main() 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-08T10:12:02.436 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: r = ctx.func(ctx) 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: return func(ctx) 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: raise RuntimeError('uid/gid not found') 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: RuntimeError: uid/gid not found 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Traceback (most recent call last): 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: yield (conn, connr) 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-08T10:12:02.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: code, '\n'.join(err))) 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi023 ... 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:07:37Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:08:15Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:08:53Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi023 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:10:07Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:10:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr time="2023-12-08T10:11:22Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-08T10:12:02.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Traceback (most recent call last): 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: main() 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: r = ctx.func(ctx) 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: return func(ctx) 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: raise RuntimeError('uid/gid not found') 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: RuntimeError: uid/gid not found 2023-12-08T10:12:02.439 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:12:00.174156+0000 mgr.smithi023.tgiigg (mgr.14164) 171 : cephadm [INF] Deploying daemon crash.smithi023 on smithi023 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:00.358332+0000 mgr.smithi023.tgiigg (mgr.14164) 172 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.366194+0000 mon.smithi023 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.170:0/3550333126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.631596+0000 mon.smithi023 (mon.0) 243 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.687890+0000 mon.smithi023 (mon.0) 244 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.690373+0000 mon.smithi023 (mon.0) 245 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.690762+0000 mon.smithi023 (mon.0) 246 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-08T10:12:02.440 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.693639+0000 mon.smithi023 (mon.0) 247 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:12:02.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:02.754 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:03.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:01.691215+0000 mgr.smithi023.tgiigg (mgr.14164) 173 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-08T10:12:03.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:12:01.697048+0000 mgr.smithi023.tgiigg (mgr.14164) 174 : cephadm [INF] Deploying daemon grafana.smithi023 on smithi023 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 -- 172.21.15.170:0/2185880085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 msgr2=0x7fb6180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 --2- 172.21.15.170:0/2185880085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb6180f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fb600009fb0 tx=0x7fb600009b30).stop 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 -- 172.21.15.170:0/2185880085 shutdown_connections 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 --2- 172.21.15.170:0/2185880085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb6180f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 -- 172.21.15.170:0/2185880085 >> 172.21.15.170:0/2185880085 conn(0x7fb6180f0b70 msgr2=0x7fb6180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 -- 172.21.15.170:0/2185880085 shutdown_connections 2023-12-08T10:12:03.944 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.942+0000 7fb61d3ca700 1 -- 172.21.15.170:0/2185880085 wait complete. 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb61d3ca700 1 Processor -- start 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb61d3ca700 1 -- start start 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb61d3ca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb618106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb61d3ca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb618106c00 con 0x7fb6180f5410 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb616ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb618106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb616ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb618106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59482/0 (socket says 172.21.15.170:59482) 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.943+0000 7fb616ffd700 1 -- 172.21.15.170:0/1372919916 learned_addr learned my addr 172.21.15.170:0/1372919916 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb616ffd700 1 -- 172.21.15.170:0/1372919916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb600005040 con 0x7fb6180f5410 2023-12-08T10:12:03.946 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb616ffd700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb618106710 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb600013510 tx=0x7fb60000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb600009320 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb600013f00 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb618106e00 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb600014d90 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.944+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb618107290 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.945+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb618045ad0 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.945+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6000095f0 con 0x7fb6180f5410 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.945+0000 7fb60f7fe700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb60402ff60 0x7fb604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.945+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6000410c0 con 0x7fb6180f5410 2023-12-08T10:12:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.945+0000 7fb6167fc700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb60402ff60 0x7fb604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:03.948 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.946+0000 7fb6167fc700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb60402ff60 0x7fb604032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb6080096a0 tx=0x7fb608006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:03.951 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:03.950+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb600014420 con 0x7fb6180f5410 2023-12-08T10:12:04.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.151+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb618108140 con 0x7fb6180f5410 2023-12-08T10:12:04.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.152+0000 7fb60f7fe700 1 -- 172.21.15.170:0/1372919916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb600010230 con 0x7fb6180f5410 2023-12-08T10:12:04.154 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:04.154 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:04.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb60402ff60 msgr2=0x7fb604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:04.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb60402ff60 0x7fb604032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb6080096a0 tx=0x7fb608006b40).stop 2023-12-08T10:12:04.155 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 msgr2=0x7fb618106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb618106710 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb600013510 tx=0x7fb60000b040).stop 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 shutdown_connections 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb60402ff60 0x7fb604032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 --2- 172.21.15.170:0/1372919916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6180f5410 0x7fb618106710 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 >> 172.21.15.170:0/1372919916 conn(0x7fb6180f0b70 msgr2=0x7fb61805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 shutdown_connections 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:04.154+0000 7fb61d3ca700 1 -- 172.21.15.170:0/1372919916 wait complete. 2023-12-08T10:12:04.156 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:02.358649+0000 mgr.smithi023.tgiigg (mgr.14164) 175 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:05.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:04.152929+0000 mon.smithi023 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.170:0/1372919916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:05.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:05.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:04.358940+0000 mgr.smithi023.tgiigg (mgr.14164) 176 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:06.712 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.709+0000 7ff35b952700 1 -- 172.21.15.170:0/3286126000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 msgr2=0x7ff3540f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.709+0000 7ff35b952700 1 --2- 172.21.15.170:0/3286126000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff3540f5d70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff344009fb0 tx=0x7ff344009b30).stop 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.709+0000 7ff35b952700 1 -- 172.21.15.170:0/3286126000 shutdown_connections 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.709+0000 7ff35b952700 1 --2- 172.21.15.170:0/3286126000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff3540f5d70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.709+0000 7ff35b952700 1 -- 172.21.15.170:0/3286126000 >> 172.21.15.170:0/3286126000 conn(0x7ff3540f0260 msgr2=0x7ff3540f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.710+0000 7ff35b952700 1 -- 172.21.15.170:0/3286126000 shutdown_connections 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.710+0000 7ff35b952700 1 -- 172.21.15.170:0/3286126000 wait complete. 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.710+0000 7ff35b952700 1 Processor -- start 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.710+0000 7ff35b952700 1 -- start start 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.710+0000 7ff35b952700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff35410b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.710+0000 7ff35b952700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff354107220 con 0x7ff3540f5950 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff3596ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff35410b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff3596ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff35410b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59508/0 (socket says 172.21.15.170:59508) 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff3596ee700 1 -- 172.21.15.170:0/3156465586 learned_addr learned my addr 172.21.15.170:0/3156465586 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:06.713 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff3596ee700 1 -- 172.21.15.170:0/3156465586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344005040 con 0x7ff3540f5950 2023-12-08T10:12:06.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff3596ee700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff35410b830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff344013510 tx=0x7ff34400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:06.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff344009320 con 0x7ff3540f5950 2023-12-08T10:12:06.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3540f5310 con 0x7ff3540f5950 2023-12-08T10:12:06.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff344013f00 con 0x7ff3540f5950 2023-12-08T10:12:06.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff344014d90 con 0x7ff3540f5950 2023-12-08T10:12:06.714 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.711+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35410bd20 con 0x7ff3540f5950 2023-12-08T10:12:06.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.712+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff354045ad0 con 0x7ff3540f5950 2023-12-08T10:12:06.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.712+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3440095f0 con 0x7ff3540f5950 2023-12-08T10:12:06.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.712+0000 7ff349ffb700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff34002ff60 0x7ff340032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:06.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.713+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff344041080 con 0x7ff3540f5950 2023-12-08T10:12:06.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.713+0000 7ff358eed700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff34002ff60 0x7ff340032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:06.715 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.713+0000 7ff358eed700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff34002ff60 0x7ff340032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff3500096a0 tx=0x7ff350006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:06.718 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.717+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff34401d7f0 con 0x7ff3540f5950 2023-12-08T10:12:06.924 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.923+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff35410c4e0 con 0x7ff3540f5950 2023-12-08T10:12:06.925 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.923+0000 7ff349ffb700 1 -- 172.21.15.170:0/3156465586 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff344010230 con 0x7ff3540f5950 2023-12-08T10:12:06.925 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:06.925 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff34002ff60 msgr2=0x7ff340032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff34002ff60 0x7ff340032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7ff3500096a0 tx=0x7ff350006b40).stop 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 msgr2=0x7ff35410b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff35410b830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7ff344013510 tx=0x7ff34400b040).stop 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 shutdown_connections 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff34002ff60 0x7ff340032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 --2- 172.21.15.170:0/3156465586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3540f5950 0x7ff35410b830 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 >> 172.21.15.170:0/3156465586 conn(0x7ff3540f0260 msgr2=0x7ff3540f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 shutdown_connections 2023-12-08T10:12:06.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:06.925+0000 7ff35b952700 1 -- 172.21.15.170:0/3156465586 wait complete. 2023-12-08T10:12:06.928 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:07.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:06.924278+0000 mon.smithi023 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.170:0/3156465586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:08.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:08.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:08 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:06.359301+0000 mgr.smithi023.tgiigg (mgr.14164) 177 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 -- 172.21.15.170:0/553150089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 msgr2=0x7f64180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 --2- 172.21.15.170:0/553150089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64180f55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6408009fb0 tx=0x7f6408009b30).stop 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 -- 172.21.15.170:0/553150089 shutdown_connections 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 --2- 172.21.15.170:0/553150089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64180f55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 -- 172.21.15.170:0/553150089 >> 172.21.15.170:0/553150089 conn(0x7f64180f09a0 msgr2=0x7f64180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 -- 172.21.15.170:0/553150089 shutdown_connections 2023-12-08T10:12:09.460 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 -- 172.21.15.170:0/553150089 wait complete. 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.458+0000 7f641cc20700 1 Processor -- start 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f641cc20700 1 -- start start 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f641cc20700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64181091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f641cc20700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6418060240 con 0x7f64180f51a0 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f64177fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64181091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f64177fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64181091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59512/0 (socket says 172.21.15.170:59512) 2023-12-08T10:12:09.461 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f64177fe700 1 -- 172.21.15.170:0/3623716260 learned_addr learned my addr 172.21.15.170:0/3623716260 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f64177fe700 1 -- 172.21.15.170:0/3623716260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6408005040 con 0x7f64180f51a0 2023-12-08T10:12:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.459+0000 7f64177fe700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64181091b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6408009830 tx=0x7f640800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6408009320 con 0x7f64180f51a0 2023-12-08T10:12:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6418060440 con 0x7f64180f51a0 2023-12-08T10:12:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64080136c0 con 0x7f64180f51a0 2023-12-08T10:12:09.462 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f640801ee90 con 0x7f64180f51a0 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6418109860 con 0x7f64180f51a0 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f640801e420 con 0x7f64180f51a0 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f63fffff700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f640002ff10 0x7f64000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.460+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6418045ad0 con 0x7f64180f51a0 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.461+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64080410a0 con 0x7f64180f51a0 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.461+0000 7f6416ffd700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f640002ff10 0x7f64000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:09.463 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.461+0000 7f6416ffd700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f640002ff10 0x7f64000323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f640c0096a0 tx=0x7f640c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:09.466 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.465+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f640801c790 con 0x7f64180f51a0 2023-12-08T10:12:09.671 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.670+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f641810a0e0 con 0x7f64180f51a0 2023-12-08T10:12:09.672 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.670+0000 7f63fffff700 1 -- 172.21.15.170:0/3623716260 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f640800e2a0 con 0x7f64180f51a0 2023-12-08T10:12:09.672 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:09.672 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f640002ff10 msgr2=0x7f64000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f640002ff10 0x7f64000323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f640c0096a0 tx=0x7f640c006b40).stop 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 msgr2=0x7f64181091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64181091b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6408009830 tx=0x7f640800b040).stop 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 shutdown_connections 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f640002ff10 0x7f64000323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 --2- 172.21.15.170:0/3623716260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64180f51a0 0x7f64181091b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 >> 172.21.15.170:0/3623716260 conn(0x7f64180f09a0 msgr2=0x7f64180f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 shutdown_connections 2023-12-08T10:12:09.674 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:09.672+0000 7f641cc20700 1 -- 172.21.15.170:0/3623716260 wait complete. 2023-12-08T10:12:09.675 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:08.359667+0000 mgr.smithi023.tgiigg (mgr.14164) 178 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:09.671209+0000 mon.smithi023 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.170:0/3623716260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:11.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:11.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:12.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 -- 172.21.15.170:0/1192960653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 msgr2=0x7f41a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 --2- 172.21.15.170:0/1192960653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a40f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f4198009fb0 tx=0x7f4198009b30).stop 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 -- 172.21.15.170:0/1192960653 shutdown_connections 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 --2- 172.21.15.170:0/1192960653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a40f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 -- 172.21.15.170:0/1192960653 >> 172.21.15.170:0/1192960653 conn(0x7f41a40f1c70 msgr2=0x7f41a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 -- 172.21.15.170:0/1192960653 shutdown_connections 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 -- 172.21.15.170:0/1192960653 wait complete. 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 Processor -- start 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.208+0000 7f41aa79a700 1 -- start start 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41aa79a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41aa79a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a4106bc0 con 0x7f41a40f6510 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:12.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59516/0 (socket says 172.21.15.170:59516) 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a3fff700 1 -- 172.21.15.170:0/2667136898 learned_addr learned my addr 172.21.15.170:0/2667136898 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a3fff700 1 -- 172.21.15.170:0/2667136898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4198005040 con 0x7f41a40f6510 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a3fff700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a41066d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f4198013510 tx=0x7f419800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4198009320 con 0x7f41a40f6510 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a4106dc0 con 0x7f41a40f6510 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4198013f00 con 0x7f41a40f6510 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4198014d90 con 0x7f41a40f6510 2023-12-08T10:12:12.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.209+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a4107250 con 0x7f41a40f6510 2023-12-08T10:12:12.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.210+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4198014420 con 0x7f41a40f6510 2023-12-08T10:12:12.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.210+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a4045ad0 con 0x7f41a40f6510 2023-12-08T10:12:12.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.210+0000 7f41a0ff9700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f418c02ff60 0x7f418c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:12.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.210+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41980410c0 con 0x7f41a40f6510 2023-12-08T10:12:12.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.210+0000 7f41a37fe700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f418c02ff60 0x7f418c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:12.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.211+0000 7f41a37fe700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f418c02ff60 0x7f418c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f41940096a0 tx=0x7f4194006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:12.216 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.215+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41980095f0 con 0x7f41a40f6510 2023-12-08T10:12:12.420 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.419+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41a4108100 con 0x7f41a40f6510 2023-12-08T10:12:12.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.420+0000 7f41a0ff9700 1 -- 172.21.15.170:0/2667136898 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f419801db70 con 0x7f41a40f6510 2023-12-08T10:12:12.421 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:12.421 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f418c02ff60 msgr2=0x7f418c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f418c02ff60 0x7f418c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f41940096a0 tx=0x7f4194006b40).stop 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 msgr2=0x7f41a41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a41066d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f4198013510 tx=0x7f419800b040).stop 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 shutdown_connections 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f418c02ff60 0x7f418c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 --2- 172.21.15.170:0/2667136898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a40f6510 0x7f41a41066d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 >> 172.21.15.170:0/2667136898 conn(0x7f41a40f1c70 msgr2=0x7f41a40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 shutdown_connections 2023-12-08T10:12:12.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:12.421+0000 7f41aa79a700 1 -- 172.21.15.170:0/2667136898 wait complete. 2023-12-08T10:12:12.424 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:12.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:10.359956+0000 mgr.smithi023.tgiigg (mgr.14164) 179 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:13.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:12.420676+0000 mon.smithi023 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.170:0/2667136898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:13.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:13.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:14.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:12.360172+0000 mgr.smithi023.tgiigg (mgr.14164) 180 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:14.926 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 -- 172.21.15.170:0/1235269288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 msgr2=0x7f31280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 --2- 172.21.15.170:0/1235269288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31280f55c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f311c009fb0 tx=0x7f311c009b30).stop 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 -- 172.21.15.170:0/1235269288 shutdown_connections 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 --2- 172.21.15.170:0/1235269288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31280f55c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 -- 172.21.15.170:0/1235269288 >> 172.21.15.170:0/1235269288 conn(0x7f31280f09a0 msgr2=0x7f31280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 -- 172.21.15.170:0/1235269288 shutdown_connections 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.924+0000 7f312c8d7700 1 -- 172.21.15.170:0/1235269288 wait complete. 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f312c8d7700 1 Processor -- start 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f312c8d7700 1 -- start start 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f312c8d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31281053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f312c8d7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281058a0 con 0x7f31280f51a0 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f3126d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31281053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f3126d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31281053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:50434/0 (socket says 172.21.15.170:50434) 2023-12-08T10:12:14.927 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.925+0000 7f3126d9d700 1 -- 172.21.15.170:0/2340871705 learned_addr learned my addr 172.21.15.170:0/2340871705 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:14.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f3126d9d700 1 -- 172.21.15.170:0/2340871705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c005040 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f3126d9d700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31281053b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f311c013510 tx=0x7f311c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f311c009320 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f311c013f00 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3128105aa0 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f311c014dc0 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.926+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3128105f30 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.927+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f311c0095f0 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.927+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3128045ad0 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.927+0000 7f310f7fe700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f311002ffb0 0x7f3110032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.927+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f311c040ff0 con 0x7f31280f51a0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.927+0000 7f312659c700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f311002ffb0 0x7f3110032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:14.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.928+0000 7f312659c700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f311002ffb0 0x7f3110032470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f31180096a0 tx=0x7f3118006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:14.933 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:14.932+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f311c06d2b0 con 0x7f31280f51a0 2023-12-08T10:12:15.141 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.140+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3128106de0 con 0x7f31280f51a0 2023-12-08T10:12:15.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.141+0000 7f310f7fe700 1 -- 172.21.15.170:0/2340871705 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f311c014420 con 0x7f31280f51a0 2023-12-08T10:12:15.142 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:15.142 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f311002ffb0 msgr2=0x7f3110032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f311002ffb0 0x7f3110032470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f31180096a0 tx=0x7f3118006b40).stop 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 msgr2=0x7f31281053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31281053b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f311c013510 tx=0x7f311c00b040).stop 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 shutdown_connections 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f311002ffb0 0x7f3110032470 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 --2- 172.21.15.170:0/2340871705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31280f51a0 0x7f31281053b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 >> 172.21.15.170:0/2340871705 conn(0x7f31280f09a0 msgr2=0x7f31280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 shutdown_connections 2023-12-08T10:12:15.144 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:15.142+0000 7f312c8d7700 1 -- 172.21.15.170:0/2340871705 wait complete. 2023-12-08T10:12:15.145 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:16.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:14.360386+0000 mgr.smithi023.tgiigg (mgr.14164) 181 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:16.395 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:15.141609+0000 mon.smithi023 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.170:0/2340871705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:16.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:16.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:17.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 -- 172.21.15.170:0/2685455374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 msgr2=0x7f0d2c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:17.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 --2- 172.21.15.170:0/2685455374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f8f20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f0d20009fb0 tx=0x7f0d20009b30).stop 2023-12-08T10:12:17.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 -- 172.21.15.170:0/2685455374 shutdown_connections 2023-12-08T10:12:17.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 --2- 172.21.15.170:0/2685455374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c0f8f20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:17.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 -- 172.21.15.170:0/2685455374 >> 172.21.15.170:0/2685455374 conn(0x7f0d2c0f09a0 msgr2=0x7f0d2c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:17.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 -- 172.21.15.170:0/2685455374 shutdown_connections 2023-12-08T10:12:17.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 -- 172.21.15.170:0/2685455374 wait complete. 2023-12-08T10:12:17.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 Processor -- start 2023-12-08T10:12:17.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.608+0000 7f0d2bfff700 1 -- start start 2023-12-08T10:12:17.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:17.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c1069f0 con 0x7f0d2c0601e0 2023-12-08T10:12:17.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:50462/0 (socket says 172.21.15.170:50462) 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2affd700 1 -- 172.21.15.170:0/3961758344 learned_addr learned my addr 172.21.15.170:0/3961758344 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2affd700 1 -- 172.21.15.170:0/3961758344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20005040 con 0x7f0d2c0601e0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2affd700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c106500 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f0d20013510 tx=0x7f0d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d20009320 con 0x7f0d2c0601e0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d20013f00 con 0x7f0d2c0601e0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2c106bf0 con 0x7f0d2c0601e0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.609+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d20014d90 con 0x7f0d2c0601e0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.610+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2c107080 con 0x7f0d2c0601e0 2023-12-08T10:12:17.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.610+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d200263a0 con 0x7f0d2c0601e0 2023-12-08T10:12:17.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.610+0000 7f0d137fe700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0d1402ff60 0x7f0d14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:17.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.611+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d200413b0 con 0x7f0d2c0601e0 2023-12-08T10:12:17.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.610+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2c045ad0 con 0x7f0d2c0601e0 2023-12-08T10:12:17.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.611+0000 7f0d2a7fc700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0d1402ff60 0x7f0d14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:17.614 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.611+0000 7f0d2a7fc700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0d1402ff60 0x7f0d14032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0d1c0096a0 tx=0x7f0d1c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:17.617 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.615+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d20012920 con 0x7f0d2c0601e0 2023-12-08T10:12:17.824 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.822+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d2c107f30 con 0x7f0d2c0601e0 2023-12-08T10:12:17.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.823+0000 7f0d137fe700 1 -- 172.21.15.170:0/3961758344 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d2002b030 con 0x7f0d2c0601e0 2023-12-08T10:12:17.825 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:17.825 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0d1402ff60 msgr2=0x7f0d14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0d1402ff60 0x7f0d14032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0d1c0096a0 tx=0x7f0d1c006b40).stop 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 msgr2=0x7f0d2c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c106500 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f0d20013510 tx=0x7f0d2000b040).stop 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 shutdown_connections 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0d1402ff60 0x7f0d14032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 --2- 172.21.15.170:0/3961758344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0d2c0601e0 0x7f0d2c106500 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 >> 172.21.15.170:0/3961758344 conn(0x7f0d2c0f09a0 msgr2=0x7f0d2c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 shutdown_connections 2023-12-08T10:12:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:17.824+0000 7f0d2bfff700 1 -- 172.21.15.170:0/3961758344 wait complete. 2023-12-08T10:12:17.828 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:18.353 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:16.360585+0000 mgr.smithi023.tgiigg (mgr.14164) 182 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:18.354 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:17.824227+0000 mon.smithi023 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.170:0/3961758344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:19.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:19.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:19.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:18.104515+0000 mon.smithi023 (mon.0) 254 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' 2023-12-08T10:12:19.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:12:18.105853+0000 mgr.smithi023.tgiigg (mgr.14164) 183 : cephadm [INF] Deploying daemon node-exporter.smithi023 on smithi023 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- 172.21.15.170:0/1418198465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 msgr2=0x7fd3f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 --2- 172.21.15.170:0/1418198465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f00f55c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fd3ec009fb0 tx=0x7fd3ec009b30).stop 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- 172.21.15.170:0/1418198465 shutdown_connections 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 --2- 172.21.15.170:0/1418198465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f00f55c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- 172.21.15.170:0/1418198465 >> 172.21.15.170:0/1418198465 conn(0x7fd3f00f09a0 msgr2=0x7fd3f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- 172.21.15.170:0/1418198465 shutdown_connections 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- 172.21.15.170:0/1418198465 wait complete. 2023-12-08T10:12:20.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 Processor -- start 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- start start 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.368+0000 7fd3f608e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01069d0 con 0x7fd3f00f51a0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f508c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f508c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:50478/0 (socket says 172.21.15.170:50478) 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f508c700 1 -- 172.21.15.170:0/3326870119 learned_addr learned my addr 172.21.15.170:0/3326870119 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f508c700 1 -- 172.21.15.170:0/3326870119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec005040 con 0x7fd3f00f51a0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f508c700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f01064e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fd3ec013510 tx=0x7fd3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3ec009320 con 0x7fd3f00f51a0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f0106bd0 con 0x7fd3f00f51a0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3ec013f00 con 0x7fd3f00f51a0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3ec014dc0 con 0x7fd3f00f51a0 2023-12-08T10:12:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.369+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f0107060 con 0x7fd3f00f51a0 2023-12-08T10:12:20.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.370+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3ec0095f0 con 0x7fd3f00f51a0 2023-12-08T10:12:20.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.370+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f0045ad0 con 0x7fd3f00f51a0 2023-12-08T10:12:20.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.370+0000 7fd3e5ffb700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd3dc02ff60 0x7fd3dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:20.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.370+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3ec041020 con 0x7fd3f00f51a0 2023-12-08T10:12:20.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.371+0000 7fd3f488b700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd3dc02ff60 0x7fd3dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:20.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.371+0000 7fd3f488b700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd3dc02ff60 0x7fd3dc032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fd3e00096a0 tx=0x7fd3e0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:20.378 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.375+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3ec014420 con 0x7fd3f00f51a0 2023-12-08T10:12:20.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:18.360769+0000 mgr.smithi023.tgiigg (mgr.14164) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:20.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.580+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3f0107eb0 con 0x7fd3f00f51a0 2023-12-08T10:12:20.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.580+0000 7fd3e5ffb700 1 -- 172.21.15.170:0/3326870119 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3ec01c920 con 0x7fd3f00f51a0 2023-12-08T10:12:20.583 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:20.583 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd3dc02ff60 msgr2=0x7fd3dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd3dc02ff60 0x7fd3dc032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fd3e00096a0 tx=0x7fd3e0006b40).stop 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 msgr2=0x7fd3f01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f01064e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fd3ec013510 tx=0x7fd3ec00b040).stop 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 shutdown_connections 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd3dc02ff60 0x7fd3dc032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 --2- 172.21.15.170:0/3326870119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3f00f51a0 0x7fd3f01064e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:20.585 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 >> 172.21.15.170:0/3326870119 conn(0x7fd3f00f09a0 msgr2=0x7fd3f005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:20.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 shutdown_connections 2023-12-08T10:12:20.586 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:20.582+0000 7fd3f608e700 1 -- 172.21.15.170:0/3326870119 wait complete. 2023-12-08T10:12:20.586 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:21.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:20.582224+0000 mon.smithi023 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.170:0/3326870119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:21.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:21.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:22.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:20.361026+0000 mgr.smithi023.tgiigg (mgr.14164) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 -- 172.21.15.170:0/1302885493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 msgr2=0x7f58bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 --2- 172.21.15.170:0/1302885493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc0f6930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f58b0009fb0 tx=0x7f58b0009b30).stop 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 -- 172.21.15.170:0/1302885493 shutdown_connections 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 --2- 172.21.15.170:0/1302885493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc0f6930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 -- 172.21.15.170:0/1302885493 >> 172.21.15.170:0/1302885493 conn(0x7f58bc0f1c70 msgr2=0x7f58bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 -- 172.21.15.170:0/1302885493 shutdown_connections 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.099+0000 7f58c2127700 1 -- 172.21.15.170:0/1302885493 wait complete. 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58c2127700 1 Processor -- start 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58c2127700 1 -- start start 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58c2127700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:23.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58c2127700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc106bc0 con 0x7f58bc0f6510 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58bb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58bb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:46422/0 (socket says 172.21.15.170:46422) 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.100+0000 7f58bb7fe700 1 -- 172.21.15.170:0/2502777277 learned_addr learned my addr 172.21.15.170:0/2502777277 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58bb7fe700 1 -- 172.21.15.170:0/2502777277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b0005040 con 0x7f58bc0f6510 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58bb7fe700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc1066d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f58b0014000 tx=0x7f58b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58b00095f0 con 0x7f58bc0f6510 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58b0007e70 con 0x7f58bc0f6510 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc106dc0 con 0x7f58bc0f6510 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58b0014ea0 con 0x7f58bc0f6510 2023-12-08T10:12:23.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.101+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc107250 con 0x7f58bc0f6510 2023-12-08T10:12:23.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.102+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58b0025070 con 0x7f58bc0f6510 2023-12-08T10:12:23.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.102+0000 7f58a3fff700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f58a402ff60 0x7f58a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:23.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.102+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58b0040a10 con 0x7f58bc0f6510 2023-12-08T10:12:23.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.102+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc045ad0 con 0x7f58bc0f6510 2023-12-08T10:12:23.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.102+0000 7f58baffd700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f58a402ff60 0x7f58a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:23.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.103+0000 7f58baffd700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f58a402ff60 0x7f58a4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f58ac0096a0 tx=0x7f58ac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:23.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.107+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f58b0026350 con 0x7f58bc0f6510 2023-12-08T10:12:23.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.314+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58bc107f70 con 0x7f58bc0f6510 2023-12-08T10:12:23.317 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.315+0000 7f58a3fff700 1 -- 172.21.15.170:0/2502777277 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58b0012330 con 0x7f58bc0f6510 2023-12-08T10:12:23.317 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:23.318 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:23.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f58a402ff60 msgr2=0x7f58a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:23.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f58a402ff60 0x7f58a4032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f58ac0096a0 tx=0x7f58ac006b40).stop 2023-12-08T10:12:23.319 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 msgr2=0x7f58bc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc1066d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f58b0014000 tx=0x7f58b000b040).stop 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 shutdown_connections 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f58a402ff60 0x7f58a4032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 --2- 172.21.15.170:0/2502777277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58bc0f6510 0x7f58bc1066d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 >> 172.21.15.170:0/2502777277 conn(0x7f58bc0f1c70 msgr2=0x7f58bc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 shutdown_connections 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:23.317+0000 7f58c2127700 1 -- 172.21.15.170:0/2502777277 wait complete. 2023-12-08T10:12:23.320 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:22.454668+0000 mon.smithi023 (mon.0) 256 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:12:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:22.456423+0000 mon.smithi023 (mon.0) 257 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:12:24.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:22.361345+0000 mgr.smithi023.tgiigg (mgr.14164) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:24.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:23.317036+0000 mon.smithi023 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.170:0/2502777277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:24.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:24.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:25.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:24.361713+0000 mgr.smithi023.tgiigg (mgr.14164) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 -- 172.21.15.170:0/3239439956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 msgr2=0x7fa5b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 --2- 172.21.15.170:0/3239439956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b40f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa5a8009fb0 tx=0x7fa5a8009b30).stop 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 -- 172.21.15.170:0/3239439956 shutdown_connections 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 --2- 172.21.15.170:0/3239439956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b40f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 -- 172.21.15.170:0/3239439956 >> 172.21.15.170:0/3239439956 conn(0x7fa5b40f0b70 msgr2=0x7fa5b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 -- 172.21.15.170:0/3239439956 shutdown_connections 2023-12-08T10:12:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.900+0000 7fa5b8f54700 1 -- 172.21.15.170:0/3239439956 wait complete. 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b8f54700 1 Processor -- start 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b8f54700 1 -- start start 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b8f54700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b8f54700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4060240 con 0x7fa5b40f5410 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:46452/0 (socket says 172.21.15.170:46452) 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b259c700 1 -- 172.21.15.170:0/558945766 learned_addr learned my addr 172.21.15.170:0/558945766 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.901+0000 7fa5b259c700 1 -- 172.21.15.170:0/558945766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a8005040 con 0x7fa5b40f5410 2023-12-08T10:12:25.905 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.902+0000 7fa5b259c700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b4109430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa5a8014000 tx=0x7fa5a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.902+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5a80095f0 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.902+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b4060440 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.902+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa5a8007e70 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.902+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5a8014ea0 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.902+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b4109ae0 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.903+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5a8014690 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.903+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b4045ad0 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.903+0000 7fa5a2ffd700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa59c02ff60 0x7fa59c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.903+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5a8040870 con 0x7fa5b40f5410 2023-12-08T10:12:25.906 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.903+0000 7fa5b1d9b700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa59c02ff60 0x7fa59c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:25.907 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.903+0000 7fa5b1d9b700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa59c02ff60 0x7fa59c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa5a40096a0 tx=0x7fa5a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:25.909 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:25.907+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5a800f300 con 0x7fa5b40f5410 2023-12-08T10:12:26.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.115+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5b40fae60 con 0x7fa5b40f5410 2023-12-08T10:12:26.117 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.115+0000 7fa5a2ffd700 1 -- 172.21.15.170:0/558945766 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa5a800f300 con 0x7fa5b40f5410 2023-12-08T10:12:26.118 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:26.118 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:26.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa59c02ff60 msgr2=0x7fa59c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:26.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa59c02ff60 0x7fa59c032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fa5a40096a0 tx=0x7fa5a4006b40).stop 2023-12-08T10:12:26.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 msgr2=0x7fa5b4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:26.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b4109430 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa5a8014000 tx=0x7fa5a800b040).stop 2023-12-08T10:12:26.119 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 shutdown_connections 2023-12-08T10:12:26.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa59c02ff60 0x7fa59c032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:26.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 --2- 172.21.15.170:0/558945766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5b40f5410 0x7fa5b4109430 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:26.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 >> 172.21.15.170:0/558945766 conn(0x7fa5b40f0b70 msgr2=0x7fa5b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:26.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 shutdown_connections 2023-12-08T10:12:26.120 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:26.117+0000 7fa5b8f54700 1 -- 172.21.15.170:0/558945766 wait complete. 2023-12-08T10:12:26.120 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:26.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:26 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:26.117150+0000 mon.smithi023 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.170:0/558945766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:27.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:27.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:27.682 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:26.362103+0000 mgr.smithi023.tgiigg (mgr.14164) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:28.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 -- 172.21.15.170:0/4172365747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 msgr2=0x7f61880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:28.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 --2- 172.21.15.170:0/4172365747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f61880f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f617c009fb0 tx=0x7f617c009b30).stop 2023-12-08T10:12:28.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 -- 172.21.15.170:0/4172365747 shutdown_connections 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 --2- 172.21.15.170:0/4172365747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f61880f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 -- 172.21.15.170:0/4172365747 >> 172.21.15.170:0/4172365747 conn(0x7f61880f0b70 msgr2=0x7f61880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 -- 172.21.15.170:0/4172365747 shutdown_connections 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 -- 172.21.15.170:0/4172365747 wait complete. 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.698+0000 7f618dbfa700 1 Processor -- start 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f618dbfa700 1 -- start start 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f618dbfa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f6188105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f618dbfa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6188105b20 con 0x7f61880f5410 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f61877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f6188105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f61877fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f6188105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:46472/0 (socket says 172.21.15.170:46472) 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f61877fe700 1 -- 172.21.15.170:0/522645255 learned_addr learned my addr 172.21.15.170:0/522645255 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f61877fe700 1 -- 172.21.15.170:0/522645255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c005040 con 0x7f61880f5410 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.699+0000 7f61877fe700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f6188105630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f617c013510 tx=0x7f617c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:28.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.700+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617c009320 con 0x7f61880f5410 2023-12-08T10:12:28.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.700+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6188105d20 con 0x7f61880f5410 2023-12-08T10:12:28.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.700+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f617c013f00 con 0x7f61880f5410 2023-12-08T10:12:28.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.700+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617c014d90 con 0x7f61880f5410 2023-12-08T10:12:28.704 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.700+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61881061b0 con 0x7f61880f5410 2023-12-08T10:12:28.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.700+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f617c0263a0 con 0x7f61880f5410 2023-12-08T10:12:28.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.701+0000 7f616ffff700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f617002ff60 0x7f6170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:28.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.701+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f617c041030 con 0x7f61880f5410 2023-12-08T10:12:28.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.701+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6188045ad0 con 0x7f61880f5410 2023-12-08T10:12:28.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.701+0000 7f6186ffd700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f617002ff60 0x7f6170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:28.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.701+0000 7f6186ffd700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f617002ff60 0x7f6170032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f61780096a0 tx=0x7f6178006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:28.708 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.705+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f617c012920 con 0x7f61880f5410 2023-12-08T10:12:28.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.910+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61880fae60 con 0x7f61880f5410 2023-12-08T10:12:28.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.910+0000 7f616ffff700 1 -- 172.21.15.170:0/522645255 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f617c02b020 con 0x7f61880f5410 2023-12-08T10:12:28.913 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:28.913 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f617002ff60 msgr2=0x7f6170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f617002ff60 0x7f6170032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f61780096a0 tx=0x7f6178006b40).stop 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 msgr2=0x7f6188105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f6188105630 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f617c013510 tx=0x7f617c00b040).stop 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 shutdown_connections 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f617002ff60 0x7f6170032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 --2- 172.21.15.170:0/522645255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61880f5410 0x7f6188105630 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 >> 172.21.15.170:0/522645255 conn(0x7f61880f0b70 msgr2=0x7f61880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 shutdown_connections 2023-12-08T10:12:28.915 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:28.912+0000 7f618dbfa700 1 -- 172.21.15.170:0/522645255 wait complete. 2023-12-08T10:12:28.916 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:28.362463+0000 mgr.smithi023.tgiigg (mgr.14164) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:28.912304+0000 mon.smithi023 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.170:0/522645255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:30.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:30.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:31.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 -- 172.21.15.170:0/4235990802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 msgr2=0x7f22280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:31.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 --2- 172.21.15.170:0/4235990802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f22280f66c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f221c009fb0 tx=0x7f221c009b30).stop 2023-12-08T10:12:31.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 -- 172.21.15.170:0/4235990802 shutdown_connections 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 --2- 172.21.15.170:0/4235990802 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f22280f66c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 -- 172.21.15.170:0/4235990802 >> 172.21.15.170:0/4235990802 conn(0x7f22280f1aa0 msgr2=0x7f22280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 -- 172.21.15.170:0/4235990802 shutdown_connections 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 -- 172.21.15.170:0/4235990802 wait complete. 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.466+0000 7f222d82d700 1 Processor -- start 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222d82d700 1 -- start start 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222d82d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f222806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222d82d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228069e20 con 0x7f22280f62a0 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222c82b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f222806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222c82b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f222806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:46482/0 (socket says 172.21.15.170:46482) 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222c82b700 1 -- 172.21.15.170:0/3111164059 learned_addr learned my addr 172.21.15.170:0/3111164059 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.467+0000 7f222c82b700 1 -- 172.21.15.170:0/3111164059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c005040 con 0x7f22280f62a0 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.468+0000 7f222c82b700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f222806b9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f221c013510 tx=0x7f221c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:31.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.468+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221c009320 con 0x7f22280f62a0 2023-12-08T10:12:31.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.468+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2228069fc0 con 0x7f22280f62a0 2023-12-08T10:12:31.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.468+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f221c013f00 con 0x7f22280f62a0 2023-12-08T10:12:31.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.468+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f221c014dc0 con 0x7f22280f62a0 2023-12-08T10:12:31.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.468+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222806a450 con 0x7f22280f62a0 2023-12-08T10:12:31.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.469+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f221c014420 con 0x7f22280f62a0 2023-12-08T10:12:31.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.469+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2228059100 con 0x7f22280f62a0 2023-12-08T10:12:31.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.469+0000 7f22257fa700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f221802ff60 0x7f2218032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:31.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.469+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f221c041020 con 0x7f22280f62a0 2023-12-08T10:12:31.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.469+0000 7f2227fff700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f221802ff60 0x7f2218032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:31.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.469+0000 7f2227fff700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f221802ff60 0x7f2218032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f22140096a0 tx=0x7f2214006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:31.475 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.473+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f221c0095f0 con 0x7f22280f62a0 2023-12-08T10:12:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.679+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2228045ad0 con 0x7f22280f62a0 2023-12-08T10:12:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.679+0000 7f22257fa700 1 -- 172.21.15.170:0/3111164059 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f221c01cb10 con 0x7f22280f62a0 2023-12-08T10:12:31.682 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:31.682 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f221802ff60 msgr2=0x7f2218032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f221802ff60 0x7f2218032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f22140096a0 tx=0x7f2214006b40).stop 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 msgr2=0x7f222806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f222806b9a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f221c013510 tx=0x7f221c00b040).stop 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 shutdown_connections 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f221802ff60 0x7f2218032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:31.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 --2- 172.21.15.170:0/3111164059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22280f62a0 0x7f222806b9a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 >> 172.21.15.170:0/3111164059 conn(0x7f22280f1aa0 msgr2=0x7f22280f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 shutdown_connections 2023-12-08T10:12:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:31.681+0000 7f222d82d700 1 -- 172.21.15.170:0/3111164059 wait complete. 2023-12-08T10:12:31.684 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:30.362822+0000 mgr.smithi023.tgiigg (mgr.14164) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:31.681017+0000 mon.smithi023 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.170:0/3111164059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:33.073 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:33.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:34.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:32.363184+0000 mgr.smithi023.tgiigg (mgr.14164) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:34.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 -- 172.21.15.170:0/3146932483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 msgr2=0x7f61e00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 --2- 172.21.15.170:0/3146932483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e00f90f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f61d4009fb0 tx=0x7f61d4009b30).stop 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 -- 172.21.15.170:0/3146932483 shutdown_connections 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 --2- 172.21.15.170:0/3146932483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e00f90f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 -- 172.21.15.170:0/3146932483 >> 172.21.15.170:0/3146932483 conn(0x7f61e00f0b70 msgr2=0x7f61e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 -- 172.21.15.170:0/3146932483 shutdown_connections 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.207+0000 7f61e5468700 1 -- 172.21.15.170:0/3146932483 wait complete. 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61e5468700 1 Processor -- start 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61e5468700 1 -- start start 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61e5468700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e01066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61e5468700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0106bb0 con 0x7f61e0060280 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61deffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e01066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61deffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e01066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:49298/0 (socket says 172.21.15.170:49298) 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61deffd700 1 -- 172.21.15.170:0/1205076607 learned_addr learned my addr 172.21.15.170:0/1205076607 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:34.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61deffd700 1 -- 172.21.15.170:0/1205076607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d4005040 con 0x7f61e0060280 2023-12-08T10:12:34.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.208+0000 7f61deffd700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e01066c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f61d4013510 tx=0x7f61d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.209+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d4009320 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.209+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e0106db0 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.209+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61d4013f00 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.209+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61d4014dc0 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.209+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e0107240 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.210+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61d40095f0 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.210+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e0045ad0 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.210+0000 7f61c77fe700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.210+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61d4041080 con 0x7f61e0060280 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.210+0000 7f61de7fc700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:34.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.210+0000 7f61de7fc700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f61c802ff60 0x7f61c8032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f61d00096a0 tx=0x7f61d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:34.217 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.215+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61d401c790 con 0x7f61e0060280 2023-12-08T10:12:34.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.420+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61e01080f0 con 0x7f61e0060280 2023-12-08T10:12:34.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.421+0000 7f61c77fe700 1 -- 172.21.15.170:0/1205076607 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f61d4010230 con 0x7f61e0060280 2023-12-08T10:12:34.423 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:34.424 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f61c802ff60 msgr2=0x7f61c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f61c802ff60 0x7f61c8032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f61d00096a0 tx=0x7f61d0006b40).stop 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 msgr2=0x7f61e01066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e01066c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f61d4013510 tx=0x7f61d400b040).stop 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 shutdown_connections 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f61c802ff60 0x7f61c8032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 --2- 172.21.15.170:0/1205076607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61e0060280 0x7f61e01066c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 >> 172.21.15.170:0/1205076607 conn(0x7f61e00f0b70 msgr2=0x7f61e00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:34.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 shutdown_connections 2023-12-08T10:12:34.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:34.423+0000 7f61e5468700 1 -- 172.21.15.170:0/1205076607 wait complete. 2023-12-08T10:12:34.426 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:35.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:34 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:34.422854+0000 mon.smithi023 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.170:0/1205076607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:35.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:35.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:36.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:34.363574+0000 mgr.smithi023.tgiigg (mgr.14164) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 -- 172.21.15.170:0/2208806719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 msgr2=0x7f1a600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 --2- 172.21.15.170:0/2208806719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a600f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f1a50009fb0 tx=0x7f1a50009b30).stop 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 -- 172.21.15.170:0/2208806719 shutdown_connections 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 --2- 172.21.15.170:0/2208806719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a600f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 -- 172.21.15.170:0/2208806719 >> 172.21.15.170:0/2208806719 conn(0x7f1a600f1c70 msgr2=0x7f1a600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 -- 172.21.15.170:0/2208806719 shutdown_connections 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.981+0000 7f1a672a9700 1 -- 172.21.15.170:0/2208806719 wait complete. 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a672a9700 1 Processor -- start 2023-12-08T10:12:36.984 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a672a9700 1 -- start start 2023-12-08T10:12:36.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a672a9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a60106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:36.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a672a9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a60106c20 con 0x7f1a600f6510 2023-12-08T10:12:36.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a65045700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a60106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:36.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a65045700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a60106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:49316/0 (socket says 172.21.15.170:49316) 2023-12-08T10:12:36.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a65045700 1 -- 172.21.15.170:0/2716332047 learned_addr learned my addr 172.21.15.170:0/2716332047 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:36.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.982+0000 7f1a65045700 1 -- 172.21.15.170:0/2716332047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50005040 con 0x7f1a600f6510 2023-12-08T10:12:36.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.983+0000 7f1a65045700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a60106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f1a50013510 tx=0x7f1a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.983+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a50009320 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.983+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a60106e20 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.983+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a50013f00 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.983+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a50014d90 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.983+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a601072b0 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.984+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a500263a0 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.984+0000 7f1a55ffb700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1a4c02ff60 0x7f1a4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.984+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a500410b0 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.984+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a60045ad0 con 0x7f1a600f6510 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.984+0000 7f1a64844700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1a4c02ff60 0x7f1a4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:36.987 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.984+0000 7f1a64844700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1a4c02ff60 0x7f1a4c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1a5c0096a0 tx=0x7f1a5c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:36.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:36.989+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a50012920 con 0x7f1a600f6510 2023-12-08T10:12:37.199 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.197+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a60107fd0 con 0x7f1a600f6510 2023-12-08T10:12:37.200 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.198+0000 7f1a55ffb700 1 -- 172.21.15.170:0/2716332047 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a5002b020 con 0x7f1a600f6510 2023-12-08T10:12:37.200 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:37.200 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1a4c02ff60 msgr2=0x7f1a4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1a4c02ff60 0x7f1a4c032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f1a5c0096a0 tx=0x7f1a5c006b40).stop 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 msgr2=0x7f1a60106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a60106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f1a50013510 tx=0x7f1a5000b040).stop 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 shutdown_connections 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1a4c02ff60 0x7f1a4c032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 --2- 172.21.15.170:0/2716332047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a600f6510 0x7f1a60106730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 >> 172.21.15.170:0/2716332047 conn(0x7f1a600f1c70 msgr2=0x7f1a600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:37.202 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 shutdown_connections 2023-12-08T10:12:37.203 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:37.199+0000 7f1a672a9700 1 -- 172.21.15.170:0/2716332047 wait complete. 2023-12-08T10:12:37.203 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:38.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:36.363942+0000 mgr.smithi023.tgiigg (mgr.14164) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:38.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:37.199515+0000 mon.smithi023 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.170:0/2716332047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:38.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:38.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:39.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 -- 172.21.15.170:0/1605690125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 msgr2=0x7f418c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:39.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 --2- 172.21.15.170:0/1605690125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4180009fb0 tx=0x7f4180009b30).stop 2023-12-08T10:12:39.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 -- 172.21.15.170:0/1605690125 shutdown_connections 2023-12-08T10:12:39.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 --2- 172.21.15.170:0/1605690125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:39.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 -- 172.21.15.170:0/1605690125 >> 172.21.15.170:0/1605690125 conn(0x7f418c0f0b70 msgr2=0x7f418c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:39.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 -- 172.21.15.170:0/1605690125 shutdown_connections 2023-12-08T10:12:39.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 -- 172.21.15.170:0/1605690125 wait complete. 2023-12-08T10:12:39.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 Processor -- start 2023-12-08T10:12:39.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.696+0000 7f4193263700 1 -- start start 2023-12-08T10:12:39.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.697+0000 7f4193263700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:39.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.697+0000 7f4193263700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c060220 con 0x7f418c0f5410 2023-12-08T10:12:39.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.697+0000 7f4190fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.697+0000 7f4190fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:49346/0 (socket says 172.21.15.170:49346) 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.697+0000 7f4190fff700 1 -- 172.21.15.170:0/940488232 learned_addr learned my addr 172.21.15.170:0/940488232 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.697+0000 7f4190fff700 1 -- 172.21.15.170:0/940488232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4180005040 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.698+0000 7f4190fff700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c109450 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4180013510 tx=0x7f418000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.698+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4180009320 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.698+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c060420 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.698+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4180013f00 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.698+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4180014d90 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.698+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c109b00 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.699+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41800095f0 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.699+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c059100 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.699+0000 7f41897fa700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f417c02ff60 0x7f417c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.699+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4180040fe0 con 0x7f418c0f5410 2023-12-08T10:12:39.701 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.699+0000 7f418bfff700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f417c02ff60 0x7f417c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:39.702 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.700+0000 7f418bfff700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f417c02ff60 0x7f417c032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f41780096a0 tx=0x7f4178006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:39.705 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.704+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f418001d7f0 con 0x7f418c0f5410 2023-12-08T10:12:39.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.910+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f418c045ad0 con 0x7f418c0f5410 2023-12-08T10:12:39.912 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.911+0000 7f41897fa700 1 -- 172.21.15.170:0/940488232 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4180010230 con 0x7f418c0f5410 2023-12-08T10:12:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:39.912 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:39.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f417c02ff60 msgr2=0x7f417c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f417c02ff60 0x7f417c032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f41780096a0 tx=0x7f4178006b40).stop 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 msgr2=0x7f418c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c109450 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4180013510 tx=0x7f418000b040).stop 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 shutdown_connections 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f417c02ff60 0x7f417c032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 --2- 172.21.15.170:0/940488232 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f418c0f5410 0x7f418c109450 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 >> 172.21.15.170:0/940488232 conn(0x7f418c0f0b70 msgr2=0x7f418c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 shutdown_connections 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:39.912+0000 7f4193263700 1 -- 172.21.15.170:0/940488232 wait complete. 2023-12-08T10:12:39.914 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:40.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:38.364273+0000 mgr.smithi023.tgiigg (mgr.14164) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:41.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:39.911518+0000 mon.smithi023 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.170:0/940488232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:41.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:41.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:42.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:40.364631+0000 mgr.smithi023.tgiigg (mgr.14164) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 -- 172.21.15.170:0/1919789922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 msgr2=0x7f66840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 --2- 172.21.15.170:0/1919789922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f66840f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f6680009fb0 tx=0x7f6680009b30).stop 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 -- 172.21.15.170:0/1919789922 shutdown_connections 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 --2- 172.21.15.170:0/1919789922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f66840f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 -- 172.21.15.170:0/1919789922 >> 172.21.15.170:0/1919789922 conn(0x7f66840f0b70 msgr2=0x7f66840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 -- 172.21.15.170:0/1919789922 shutdown_connections 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.391+0000 7f668c306700 1 -- 172.21.15.170:0/1919789922 wait complete. 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668c306700 1 Processor -- start 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668c306700 1 -- start start 2023-12-08T10:12:42.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668c306700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f6684106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668c306700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6684106c00 con 0x7f66840f5410 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668a0a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f6684106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668a0a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f6684106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34272/0 (socket says 172.21.15.170:34272) 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668a0a2700 1 -- 172.21.15.170:0/2203223255 learned_addr learned my addr 172.21.15.170:0/2203223255 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668a0a2700 1 -- 172.21.15.170:0/2203223255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680005040 con 0x7f66840f5410 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.392+0000 7f668a0a2700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f6684106710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f6680014000 tx=0x7f668000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.393+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66800095f0 con 0x7f66840f5410 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.393+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6684106e00 con 0x7f66840f5410 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.393+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6680007e70 con 0x7f66840f5410 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.393+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6680014ea0 con 0x7f66840f5410 2023-12-08T10:12:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.393+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6684107290 con 0x7f66840f5410 2023-12-08T10:12:42.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.393+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6684045ad0 con 0x7f66840f5410 2023-12-08T10:12:42.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.394+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6680025070 con 0x7f66840f5410 2023-12-08T10:12:42.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.394+0000 7f667affd700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f667002ff60 0x7f6670032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:42.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.394+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6680040980 con 0x7f66840f5410 2023-12-08T10:12:42.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.394+0000 7f66898a1700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f667002ff60 0x7f6670032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:42.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.394+0000 7f66898a1700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f667002ff60 0x7f6670032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f66740096a0 tx=0x7f6674006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:42.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.398+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66800079c0 con 0x7f66840f5410 2023-12-08T10:12:42.608 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.607+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66840fae60 con 0x7f66840f5410 2023-12-08T10:12:42.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.608+0000 7f667affd700 1 -- 172.21.15.170:0/2203223255 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f668000f300 con 0x7f66840f5410 2023-12-08T10:12:42.609 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:42.610 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:42.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f667002ff60 msgr2=0x7f6670032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f667002ff60 0x7f6670032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f66740096a0 tx=0x7f6674006b40).stop 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 msgr2=0x7f6684106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f6684106710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f6680014000 tx=0x7f668000b040).stop 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 shutdown_connections 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f667002ff60 0x7f6670032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 --2- 172.21.15.170:0/2203223255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66840f5410 0x7f6684106710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 >> 172.21.15.170:0/2203223255 conn(0x7f66840f0b70 msgr2=0x7f668405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 shutdown_connections 2023-12-08T10:12:42.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:42.610+0000 7f668c306700 1 -- 172.21.15.170:0/2203223255 wait complete. 2023-12-08T10:12:42.613 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:43.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:42 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:42.608441+0000 mon.smithi023 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.170:0/2203223255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:43.994 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:43.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:42.365033+0000 mgr.smithi023.tgiigg (mgr.14164) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:45.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 -- 172.21.15.170:0/1205973865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 msgr2=0x7f0ac40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:45.157 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 --2- 172.21.15.170:0/1205973865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac40f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f0ab8009fb0 tx=0x7f0ab8009b30).stop 2023-12-08T10:12:45.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 -- 172.21.15.170:0/1205973865 shutdown_connections 2023-12-08T10:12:45.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 --2- 172.21.15.170:0/1205973865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac40f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:45.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 -- 172.21.15.170:0/1205973865 >> 172.21.15.170:0/1205973865 conn(0x7f0ac40f0b70 msgr2=0x7f0ac40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:45.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 -- 172.21.15.170:0/1205973865 shutdown_connections 2023-12-08T10:12:45.158 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.156+0000 7f0ac94ae700 1 -- 172.21.15.170:0/1205973865 wait complete. 2023-12-08T10:12:45.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac94ae700 1 Processor -- start 2023-12-08T10:12:45.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac94ae700 1 -- start start 2023-12-08T10:12:45.159 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac94ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac94ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac4106c10 con 0x7f0ac40f5410 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34280/0 (socket says 172.21.15.170:34280) 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac2ffd700 1 -- 172.21.15.170:0/142827387 learned_addr learned my addr 172.21.15.170:0/142827387 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.157+0000 7f0ac2ffd700 1 -- 172.21.15.170:0/142827387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab8005040 con 0x7f0ac40f5410 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.158+0000 7f0ac2ffd700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac4106720 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0ab8013510 tx=0x7f0ab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.158+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab8009320 con 0x7f0ac40f5410 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.158+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac4106e10 con 0x7f0ac40f5410 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.158+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ab8013f00 con 0x7f0ac40f5410 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.158+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ab8014d90 con 0x7f0ac40f5410 2023-12-08T10:12:45.160 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.158+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac41072a0 con 0x7f0ac40f5410 2023-12-08T10:12:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.159+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ab80095f0 con 0x7f0ac40f5410 2023-12-08T10:12:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.159+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac4045ad0 con 0x7f0ac40f5410 2023-12-08T10:12:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.159+0000 7f0aab7fe700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0aac02ff60 0x7f0aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.159+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ab8041000 con 0x7f0ac40f5410 2023-12-08T10:12:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.159+0000 7f0ac27fc700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0aac02ff60 0x7f0aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:45.161 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.159+0000 7f0ac27fc700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0aac02ff60 0x7f0aac032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f0ab40096a0 tx=0x7f0ab4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:45.165 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.164+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ab8014420 con 0x7f0ac40f5410 2023-12-08T10:12:45.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.372+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ac4108150 con 0x7f0ac40f5410 2023-12-08T10:12:45.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.373+0000 7f0aab7fe700 1 -- 172.21.15.170:0/142827387 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ab8010230 con 0x7f0ac40f5410 2023-12-08T10:12:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:45.374 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0aac02ff60 msgr2=0x7f0aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0aac02ff60 0x7f0aac032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f0ab40096a0 tx=0x7f0ab4006b40).stop 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 msgr2=0x7f0ac4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac4106720 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0ab8013510 tx=0x7f0ab800b040).stop 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 shutdown_connections 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0aac02ff60 0x7f0aac032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 --2- 172.21.15.170:0/142827387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ac40f5410 0x7f0ac4106720 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:45.376 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 >> 172.21.15.170:0/142827387 conn(0x7f0ac40f0b70 msgr2=0x7f0ac405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:45.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 shutdown_connections 2023-12-08T10:12:45.377 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:45.375+0000 7f0ac94ae700 1 -- 172.21.15.170:0/142827387 wait complete. 2023-12-08T10:12:45.377 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:44.365431+0000 mgr.smithi023.tgiigg (mgr.14164) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:45.373769+0000 mon.smithi023 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.170:0/142827387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:46.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:46.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:47.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 -- 172.21.15.170:0/1384490598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 msgr2=0x7f9fb80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 --2- 172.21.15.170:0/1384490598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb80f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f9fa8009fb0 tx=0x7f9fa8009b30).stop 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 -- 172.21.15.170:0/1384490598 shutdown_connections 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 --2- 172.21.15.170:0/1384490598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb80f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 -- 172.21.15.170:0/1384490598 >> 172.21.15.170:0/1384490598 conn(0x7f9fb80f0b70 msgr2=0x7f9fb80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 -- 172.21.15.170:0/1384490598 shutdown_connections 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.881+0000 7f9fbe72a700 1 -- 172.21.15.170:0/1384490598 wait complete. 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fbe72a700 1 Processor -- start 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fbe72a700 1 -- start start 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fbe72a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb8108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fbe72a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb8108e10 con 0x7f9fb80f5410 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb8108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:47.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb8108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34304/0 (socket says 172.21.15.170:34304) 2023-12-08T10:12:47.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.882+0000 7f9fb7fff700 1 -- 172.21.15.170:0/556504340 learned_addr learned my addr 172.21.15.170:0/556504340 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fb7fff700 1 -- 172.21.15.170:0/556504340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8005040 con 0x7f9fb80f5410 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fb7fff700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb8108920 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9fa8013510 tx=0x7f9fa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa8009320 con 0x7f9fb80f5410 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9fa8013f00 con 0x7f9fb80f5410 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa8014d90 con 0x7f9fb80f5410 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb8109010 con 0x7f9fb80f5410 2023-12-08T10:12:47.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.883+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb81094a0 con 0x7f9fb80f5410 2023-12-08T10:12:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.884+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fa80095f0 con 0x7f9fb80f5410 2023-12-08T10:12:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.884+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb8045ad0 con 0x7f9fb80f5410 2023-12-08T10:12:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.884+0000 7f9fb4ff9700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9fa002ffb0 0x7f9fa0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.884+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fa8040f90 con 0x7f9fb80f5410 2023-12-08T10:12:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.884+0000 7f9fb77fe700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9fa002ffb0 0x7f9fa0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.885+0000 7f9fb77fe700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9fa002ffb0 0x7f9fa0032470 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9fac0096a0 tx=0x7f9fac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:47.890 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:47.889+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fa8014420 con 0x7f9fb80f5410 2023-12-08T10:12:48.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.098+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fb810a350 con 0x7f9fb80f5410 2023-12-08T10:12:48.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.098+0000 7f9fb4ff9700 1 -- 172.21.15.170:0/556504340 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9fa801d7f0 con 0x7f9fb80f5410 2023-12-08T10:12:48.100 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:48.100 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:48.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9fa002ffb0 msgr2=0x7f9fa0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:48.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9fa002ffb0 0x7f9fa0032470 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9fac0096a0 tx=0x7f9fac006b40).stop 2023-12-08T10:12:48.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 msgr2=0x7f9fb8108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb8108920 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9fa8013510 tx=0x7f9fa800b040).stop 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 shutdown_connections 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9fa002ffb0 0x7f9fa0032470 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 --2- 172.21.15.170:0/556504340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fb80f5410 0x7f9fb8108920 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 >> 172.21.15.170:0/556504340 conn(0x7f9fb80f0b70 msgr2=0x7f9fb805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 shutdown_connections 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:48.100+0000 7f9fbe72a700 1 -- 172.21.15.170:0/556504340 wait complete. 2023-12-08T10:12:48.102 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:48.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:46.365713+0000 mgr.smithi023.tgiigg (mgr.14164) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:49.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:48 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:48.099118+0000 mon.smithi023 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.170:0/556504340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:49.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:49.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:48.366007+0000 mgr.smithi023.tgiigg (mgr.14164) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:50.629 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.627+0000 7f12a0d13700 1 -- 172.21.15.170:0/603477322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 msgr2=0x7f129c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.627+0000 7f12a0d13700 1 --2- 172.21.15.170:0/603477322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c0f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f128c009fb0 tx=0x7f128c009b30).stop 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.627+0000 7f12a0d13700 1 -- 172.21.15.170:0/603477322 shutdown_connections 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.627+0000 7f12a0d13700 1 --2- 172.21.15.170:0/603477322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c0f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.627+0000 7f12a0d13700 1 -- 172.21.15.170:0/603477322 >> 172.21.15.170:0/603477322 conn(0x7f129c0f0b70 msgr2=0x7f129c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f12a0d13700 1 -- 172.21.15.170:0/603477322 shutdown_connections 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f12a0d13700 1 -- 172.21.15.170:0/603477322 wait complete. 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f12a0d13700 1 Processor -- start 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f12a0d13700 1 -- start start 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f12a0d13700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f12a0d13700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c106c10 con 0x7f129c0f5410 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f129a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f129a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:34320/0 (socket says 172.21.15.170:34320) 2023-12-08T10:12:50.630 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.628+0000 7f129a59c700 1 -- 172.21.15.170:0/3374986362 learned_addr learned my addr 172.21.15.170:0/3374986362 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f129a59c700 1 -- 172.21.15.170:0/3374986362 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c005040 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f129a59c700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c106720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f128c013510 tx=0x7f128c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f128c009320 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c106e10 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f128c013f00 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f128c014d90 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.629+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c1072a0 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.630+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f128c0095f0 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.630+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c045ad0 con 0x7f129c0f5410 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.630+0000 7f1292ffd700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f128802ff60 0x7f1288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:50.632 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.630+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f128c041000 con 0x7f129c0f5410 2023-12-08T10:12:50.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.630+0000 7f1299d9b700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f128802ff60 0x7f1288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:50.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.631+0000 7f1299d9b700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f128802ff60 0x7f1288032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f12840096a0 tx=0x7f1284006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:50.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.634+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f128c01d7f0 con 0x7f129c0f5410 2023-12-08T10:12:50.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.842+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f129c108150 con 0x7f129c0f5410 2023-12-08T10:12:50.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.843+0000 7f1292ffd700 1 -- 172.21.15.170:0/3374986362 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f128c010230 con 0x7f129c0f5410 2023-12-08T10:12:50.845 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:50.845 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:50.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f128802ff60 msgr2=0x7f1288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:50.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f128802ff60 0x7f1288032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f12840096a0 tx=0x7f1284006b40).stop 2023-12-08T10:12:50.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 msgr2=0x7f129c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c106720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f128c013510 tx=0x7f128c00b040).stop 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 shutdown_connections 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f128802ff60 0x7f1288032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 --2- 172.21.15.170:0/3374986362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0f5410 0x7f129c106720 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 >> 172.21.15.170:0/3374986362 conn(0x7f129c0f0b70 msgr2=0x7f129c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 shutdown_connections 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:50.845+0000 7f12a0d13700 1 -- 172.21.15.170:0/3374986362 wait complete. 2023-12-08T10:12:50.847 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:51.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:50 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:50.844044+0000 mon.smithi023 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.170:0/3374986362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:50.366349+0000 mgr.smithi023.tgiigg (mgr.14164) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:52.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:52.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 -- 172.21.15.170:0/1650359277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 msgr2=0x7fa6f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 --2- 172.21.15.170:0/1650359277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f40f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa6e8009fb0 tx=0x7fa6e8009b30).stop 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 -- 172.21.15.170:0/1650359277 shutdown_connections 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 --2- 172.21.15.170:0/1650359277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f40f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 -- 172.21.15.170:0/1650359277 >> 172.21.15.170:0/1650359277 conn(0x7fa6f40f0b70 msgr2=0x7fa6f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 -- 172.21.15.170:0/1650359277 shutdown_connections 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.365+0000 7fa6f9573700 1 -- 172.21.15.170:0/1650359277 wait complete. 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f9573700 1 Processor -- start 2023-12-08T10:12:53.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f9573700 1 -- start start 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f9573700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f9573700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4106c20 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53918/0 (socket says 172.21.15.170:53918) 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f2ffd700 1 -- 172.21.15.170:0/1554641684 learned_addr learned my addr 172.21.15.170:0/1554641684 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.366+0000 7fa6f2ffd700 1 -- 172.21.15.170:0/1554641684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e8005040 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.367+0000 7fa6f2ffd700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f4106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa6e8013510 tx=0x7fa6e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.367+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e8009320 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.367+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f4106e20 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.367+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6e8013f00 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.367+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e8014d90 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.367+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f41072b0 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.368+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6e80095f0 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.368+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f4045ad0 con 0x7fa6f40f5410 2023-12-08T10:12:53.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.368+0000 7fa6db7fe700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa6dc02ff60 0x7fa6dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:53.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.368+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6e8041000 con 0x7fa6f40f5410 2023-12-08T10:12:53.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.368+0000 7fa6f27fc700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa6dc02ff60 0x7fa6dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:53.373 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.369+0000 7fa6f27fc700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa6dc02ff60 0x7fa6dc032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa6e40096a0 tx=0x7fa6e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:53.374 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.373+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6e8014420 con 0x7fa6f40f5410 2023-12-08T10:12:53.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.580+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6f40fbf90 con 0x7fa6f40f5410 2023-12-08T10:12:53.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.580+0000 7fa6db7fe700 1 -- 172.21.15.170:0/1554641684 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6e801d7f0 con 0x7fa6f40f5410 2023-12-08T10:12:53.582 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:53.582 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:53.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa6dc02ff60 msgr2=0x7fa6dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa6dc02ff60 0x7fa6dc032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa6e40096a0 tx=0x7fa6e4006b40).stop 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 msgr2=0x7fa6f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f4106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa6e8013510 tx=0x7fa6e800b040).stop 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 shutdown_connections 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa6dc02ff60 0x7fa6dc032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 --2- 172.21.15.170:0/1554641684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6f40f5410 0x7fa6f4106730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 >> 172.21.15.170:0/1554641684 conn(0x7fa6f40f0b70 msgr2=0x7fa6f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 shutdown_connections 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:53.582+0000 7fa6f9573700 1 -- 172.21.15.170:0/1554641684 wait complete. 2023-12-08T10:12:53.584 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:52.366634+0000 mgr.smithi023.tgiigg (mgr.14164) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:53.581183+0000 mon.smithi023 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.170:0/1554641684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:54.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:54.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 -- 172.21.15.170:0/68122105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 msgr2=0x7f50180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 --2- 172.21.15.170:0/68122105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f50180f6930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f5000009fb0 tx=0x7f5000009b30).stop 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 -- 172.21.15.170:0/68122105 shutdown_connections 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 --2- 172.21.15.170:0/68122105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f50180f6930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 -- 172.21.15.170:0/68122105 >> 172.21.15.170:0/68122105 conn(0x7f50180f1c70 msgr2=0x7f50180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 -- 172.21.15.170:0/68122105 shutdown_connections 2023-12-08T10:12:56.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.102+0000 7f501d2cb700 1 -- 172.21.15.170:0/68122105 wait complete. 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f501d2cb700 1 Processor -- start 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f501d2cb700 1 -- start start 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f501d2cb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f5018106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f501d2cb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018106c20 con 0x7f50180f6510 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f5018106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f5016ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f5018106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53930/0 (socket says 172.21.15.170:53930) 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.103+0000 7f5016ffd700 1 -- 172.21.15.170:0/588968225 learned_addr learned my addr 172.21.15.170:0/588968225 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f5016ffd700 1 -- 172.21.15.170:0/588968225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5000005040 con 0x7f50180f6510 2023-12-08T10:12:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f5016ffd700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f5018106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f5000013510 tx=0x7f500000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000009320 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5018106e20 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5000013f00 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000014d90 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.104+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50181072b0 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.105+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5000014420 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.105+0000 7f500f7fe700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f500402ff60 0x7f5004032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.105+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5018045ad0 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.105+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5000041000 con 0x7f50180f6510 2023-12-08T10:12:56.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.105+0000 7f50167fc700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f500402ff60 0x7f5004032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:56.108 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.105+0000 7f50167fc700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f500402ff60 0x7f5004032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f50080096a0 tx=0x7f5008006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:56.111 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.110+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50000095f0 con 0x7f50180f6510 2023-12-08T10:12:56.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.319+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50180fc050 con 0x7f50180f6510 2023-12-08T10:12:56.321 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.320+0000 7f500f7fe700 1 -- 172.21.15.170:0/588968225 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f500002c090 con 0x7f50180f6510 2023-12-08T10:12:56.321 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:56.322 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f500402ff60 msgr2=0x7f5004032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f500402ff60 0x7f5004032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f50080096a0 tx=0x7f5008006b40).stop 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 msgr2=0x7f5018106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f5018106730 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f5000013510 tx=0x7f500000b040).stop 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 shutdown_connections 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f500402ff60 0x7f5004032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 --2- 172.21.15.170:0/588968225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50180f6510 0x7f5018106730 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 >> 172.21.15.170:0/588968225 conn(0x7f50180f1c70 msgr2=0x7f50180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 shutdown_connections 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:56.322+0000 7f501d2cb700 1 -- 172.21.15.170:0/588968225 wait complete. 2023-12-08T10:12:56.324 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:12:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:54.366854+0000 mgr.smithi023.tgiigg (mgr.14164) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:57.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:56 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:56.320853+0000 mon.smithi023 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.170:0/588968225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:12:57.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:12:57.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:12:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:56.367187+0000 mgr.smithi023.tgiigg (mgr.14164) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:12:58.866 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 -- 172.21.15.170:0/2965899689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 msgr2=0x7f96480f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 --2- 172.21.15.170:0/2965899689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96480f90f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f963c009fb0 tx=0x7f963c009b30).stop 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 -- 172.21.15.170:0/2965899689 shutdown_connections 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 --2- 172.21.15.170:0/2965899689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96480f90f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 -- 172.21.15.170:0/2965899689 >> 172.21.15.170:0/2965899689 conn(0x7f96480f0b70 msgr2=0x7f96480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 -- 172.21.15.170:0/2965899689 shutdown_connections 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.864+0000 7f964e02d700 1 -- 172.21.15.170:0/2965899689 wait complete. 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f964e02d700 1 Processor -- start 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f964e02d700 1 -- start start 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f964e02d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96481066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f964e02d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9648106bd0 con 0x7f9648060280 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f96477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96481066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f96477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96481066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53942/0 (socket says 172.21.15.170:53942) 2023-12-08T10:12:58.867 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.865+0000 7f96477fe700 1 -- 172.21.15.170:0/3899307600 learned_addr learned my addr 172.21.15.170:0/3899307600 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f96477fe700 1 -- 172.21.15.170:0/3899307600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c005040 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f96477fe700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96481066e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f963c013510 tx=0x7f963c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f963c009320 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f963c013f00 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9648106dd0 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f963c014dc0 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.866+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9648107260 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.867+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f963c014420 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.867+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9648045ad0 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.867+0000 7f962ffff700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f963002ffb0 0x7f9630032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.867+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f963c041110 con 0x7f9648060280 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.867+0000 7f9646ffd700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f963002ffb0 0x7f9630032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:12:58.869 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.868+0000 7f9646ffd700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f963002ffb0 0x7f9630032470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f96380096a0 tx=0x7f9638006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:12:58.873 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:58.872+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f963c026bf0 con 0x7f9648060280 2023-12-08T10:12:59.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.078+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9648108110 con 0x7f9648060280 2023-12-08T10:12:59.080 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.078+0000 7f962ffff700 1 -- 172.21.15.170:0/3899307600 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f963c01cb10 con 0x7f9648060280 2023-12-08T10:12:59.080 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:12:59.080 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:12:59.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f963002ffb0 msgr2=0x7f9630032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:59.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f963002ffb0 0x7f9630032470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f96380096a0 tx=0x7f9638006b40).stop 2023-12-08T10:12:59.081 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 msgr2=0x7f96481066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96481066e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f963c013510 tx=0x7f963c00b040).stop 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 shutdown_connections 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f963002ffb0 0x7f9630032470 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 --2- 172.21.15.170:0/3899307600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9648060280 0x7f96481066e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 >> 172.21.15.170:0/3899307600 conn(0x7f96480f0b70 msgr2=0x7f96480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 shutdown_connections 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:12:59.080+0000 7f964e02d700 1 -- 172.21.15.170:0/3899307600 wait complete. 2023-12-08T10:12:59.082 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:12:58.367541+0000 mgr.smithi023.tgiigg (mgr.14164) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:12:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:12:59.079245+0000 mon.smithi023 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.170:0/3899307600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:00.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:00.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.635+0000 7fa52be51700 1 -- 172.21.15.170:0/1852048157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 msgr2=0x7fa5240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.635+0000 7fa52be51700 1 --2- 172.21.15.170:0/1852048157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa5240f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fa514009fb0 tx=0x7fa514009b30).stop 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 -- 172.21.15.170:0/1852048157 shutdown_connections 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 --2- 172.21.15.170:0/1852048157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa5240f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 -- 172.21.15.170:0/1852048157 >> 172.21.15.170:0/1852048157 conn(0x7fa5240f1c70 msgr2=0x7fa5240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 -- 172.21.15.170:0/1852048157 shutdown_connections 2023-12-08T10:13:01.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 -- 172.21.15.170:0/1852048157 wait complete. 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 Processor -- start 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 -- start start 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa524106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.636+0000 7fa52be51700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa524106c20 con 0x7fa5240f6510 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa529bed700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa524106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa529bed700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa524106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53952/0 (socket says 172.21.15.170:53952) 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa529bed700 1 -- 172.21.15.170:0/3891729951 learned_addr learned my addr 172.21.15.170:0/3891729951 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa529bed700 1 -- 172.21.15.170:0/3891729951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514005040 con 0x7fa5240f6510 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa529bed700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa524106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fa514013510 tx=0x7fa51400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514009320 con 0x7fa5240f6510 2023-12-08T10:13:01.640 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa524106e20 con 0x7fa5240f6510 2023-12-08T10:13:01.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa514013f00 con 0x7fa5240f6510 2023-12-08T10:13:01.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa514014d90 con 0x7fa5240f6510 2023-12-08T10:13:01.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.637+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5241072b0 con 0x7fa5240f6510 2023-12-08T10:13:01.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.638+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5140095f0 con 0x7fa5240f6510 2023-12-08T10:13:01.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.638+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa524045ad0 con 0x7fa5240f6510 2023-12-08T10:13:01.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.639+0000 7fa51a7fc700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa51002ff60 0x7fa510032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:01.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.639+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa514040fe0 con 0x7fa5240f6510 2023-12-08T10:13:01.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.639+0000 7fa5293ec700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa51002ff60 0x7fa510032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:01.642 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.639+0000 7fa5293ec700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa51002ff60 0x7fa510032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fa5200096a0 tx=0x7fa520006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:01.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.643+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa51401d7f0 con 0x7fa5240f6510 2023-12-08T10:13:01.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.848+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5240fbf60 con 0x7fa5240f6510 2023-12-08T10:13:01.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.848+0000 7fa51a7fc700 1 -- 172.21.15.170:0/3891729951 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa514010230 con 0x7fa5240f6510 2023-12-08T10:13:01.851 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:01.852 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:01.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa51002ff60 msgr2=0x7fa510032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:01.852 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa51002ff60 0x7fa510032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fa5200096a0 tx=0x7fa520006b40).stop 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 msgr2=0x7fa524106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa524106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fa514013510 tx=0x7fa51400b040).stop 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 shutdown_connections 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa51002ff60 0x7fa510032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 --2- 172.21.15.170:0/3891729951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5240f6510 0x7fa524106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 >> 172.21.15.170:0/3891729951 conn(0x7fa5240f1c70 msgr2=0x7fa5240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 shutdown_connections 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:01.850+0000 7fa52be51700 1 -- 172.21.15.170:0/3891729951 wait complete. 2023-12-08T10:13:01.853 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:00.367893+0000 mgr.smithi023.tgiigg (mgr.14164) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:01.850215+0000 mon.smithi023 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.170:0/3891729951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:03.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:03.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:04.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 -- 172.21.15.170:0/450099704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 msgr2=0x7fbc8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 --2- 172.21.15.170:0/450099704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c0f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fbc80009fb0 tx=0x7fbc80009b30).stop 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 -- 172.21.15.170:0/450099704 shutdown_connections 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 --2- 172.21.15.170:0/450099704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c0f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 -- 172.21.15.170:0/450099704 >> 172.21.15.170:0/450099704 conn(0x7fbc8c0f1c70 msgr2=0x7fbc8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 -- 172.21.15.170:0/450099704 shutdown_connections 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.382+0000 7fbc944a0700 1 -- 172.21.15.170:0/450099704 wait complete. 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc944a0700 1 Processor -- start 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc944a0700 1 -- start start 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc944a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc944a0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c106c20 con 0x7fbc8c0f6510 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc9223c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc9223c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53696/0 (socket says 172.21.15.170:53696) 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc9223c700 1 -- 172.21.15.170:0/1424880462 learned_addr learned my addr 172.21.15.170:0/1424880462 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:04.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.383+0000 7fbc9223c700 1 -- 172.21.15.170:0/1424880462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc80005040 con 0x7fbc8c0f6510 2023-12-08T10:13:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.384+0000 7fbc9223c700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fbc80013510 tx=0x7fbc8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.384+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc80009320 con 0x7fbc8c0f6510 2023-12-08T10:13:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.384+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c106e20 con 0x7fbc8c0f6510 2023-12-08T10:13:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.384+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc80013f00 con 0x7fbc8c0f6510 2023-12-08T10:13:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.384+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc80014d90 con 0x7fbc8c0f6510 2023-12-08T10:13:04.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.384+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c1072b0 con 0x7fbc8c0f6510 2023-12-08T10:13:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.385+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc800095f0 con 0x7fbc8c0f6510 2023-12-08T10:13:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.385+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c045ad0 con 0x7fbc8c0f6510 2023-12-08T10:13:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.385+0000 7fbc7effd700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbc7802ff60 0x7fbc78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.385+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc80041000 con 0x7fbc8c0f6510 2023-12-08T10:13:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.385+0000 7fbc91a3b700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbc7802ff60 0x7fbc78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:04.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.385+0000 7fbc91a3b700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbc7802ff60 0x7fbc78032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbc880096a0 tx=0x7fbc88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:04.392 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.390+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc80014420 con 0x7fbc8c0f6510 2023-12-08T10:13:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:02.368317+0000 mgr.smithi023.tgiigg (mgr.14164) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:04.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.593+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc8c0fbf60 con 0x7fbc8c0f6510 2023-12-08T10:13:04.596 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.594+0000 7fbc7effd700 1 -- 172.21.15.170:0/1424880462 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbc80010230 con 0x7fbc8c0f6510 2023-12-08T10:13:04.596 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:04.596 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbc7802ff60 msgr2=0x7fbc78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbc7802ff60 0x7fbc78032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbc880096a0 tx=0x7fbc88006b40).stop 2023-12-08T10:13:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 msgr2=0x7fbc8c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fbc80013510 tx=0x7fbc8000b040).stop 2023-12-08T10:13:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 shutdown_connections 2023-12-08T10:13:04.598 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fbc7802ff60 0x7fbc78032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 --2- 172.21.15.170:0/1424880462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc8c0f6510 0x7fbc8c106730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 >> 172.21.15.170:0/1424880462 conn(0x7fbc8c0f1c70 msgr2=0x7fbc8c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 shutdown_connections 2023-12-08T10:13:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:04.596+0000 7fbc944a0700 1 -- 172.21.15.170:0/1424880462 wait complete. 2023-12-08T10:13:04.599 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:05.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:04.595896+0000 mon.smithi023 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.170:0/1424880462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:05.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:05.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:04.368738+0000 mgr.smithi023.tgiigg (mgr.14164) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:07.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 -- 172.21.15.170:0/1720904442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 msgr2=0x7fba200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 --2- 172.21.15.170:0/1720904442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba200f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fba14009fb0 tx=0x7fba14009b30).stop 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 -- 172.21.15.170:0/1720904442 shutdown_connections 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 --2- 172.21.15.170:0/1720904442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba200f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 -- 172.21.15.170:0/1720904442 >> 172.21.15.170:0/1720904442 conn(0x7fba200f1c70 msgr2=0x7fba200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 -- 172.21.15.170:0/1720904442 shutdown_connections 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.143+0000 7fba25c44700 1 -- 172.21.15.170:0/1720904442 wait complete. 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.144+0000 7fba25c44700 1 Processor -- start 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.144+0000 7fba25c44700 1 -- start start 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.144+0000 7fba25c44700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba20106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:07.147 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.144+0000 7fba25c44700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba20106c20 con 0x7fba200f6510 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba1f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba20106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba1f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba20106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53714/0 (socket says 172.21.15.170:53714) 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba1f7fe700 1 -- 172.21.15.170:0/1979132030 learned_addr learned my addr 172.21.15.170:0/1979132030 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba1f7fe700 1 -- 172.21.15.170:0/1979132030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba14005040 con 0x7fba200f6510 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba1f7fe700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba20106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fba14013510 tx=0x7fba1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba14009320 con 0x7fba200f6510 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba20106e20 con 0x7fba200f6510 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba14013f00 con 0x7fba200f6510 2023-12-08T10:13:07.149 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba14014d90 con 0x7fba200f6510 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.145+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba201072b0 con 0x7fba200f6510 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.146+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba140263a0 con 0x7fba200f6510 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.146+0000 7fba07fff700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fba0802ffb0 0x7fba08032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.146+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba140410e0 con 0x7fba200f6510 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.146+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20045ad0 con 0x7fba200f6510 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.147+0000 7fba1effd700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fba0802ffb0 0x7fba08032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:07.150 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.147+0000 7fba1effd700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fba0802ffb0 0x7fba08032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fba100096a0 tx=0x7fba10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:07.153 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.151+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba14012920 con 0x7fba200f6510 2023-12-08T10:13:07.360 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.358+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba20107fd0 con 0x7fba200f6510 2023-12-08T10:13:07.361 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.358+0000 7fba07fff700 1 -- 172.21.15.170:0/1979132030 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba1402b020 con 0x7fba200f6510 2023-12-08T10:13:07.361 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:07.361 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:07.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fba0802ffb0 msgr2=0x7fba08032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:07.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fba0802ffb0 0x7fba08032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fba100096a0 tx=0x7fba10006b40).stop 2023-12-08T10:13:07.362 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 msgr2=0x7fba20106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba20106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fba14013510 tx=0x7fba1400b040).stop 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 shutdown_connections 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fba0802ffb0 0x7fba08032470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 --2- 172.21.15.170:0/1979132030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba200f6510 0x7fba20106730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 >> 172.21.15.170:0/1979132030 conn(0x7fba200f1c70 msgr2=0x7fba200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 shutdown_connections 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:07.360+0000 7fba25c44700 1 -- 172.21.15.170:0/1979132030 wait complete. 2023-12-08T10:13:07.363 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:06.369158+0000 mgr.smithi023.tgiigg (mgr.14164) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:07.360217+0000 mon.smithi023 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.170:0/1979132030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:08.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:08.738 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:09.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 -- 172.21.15.170:0/694187657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 msgr2=0x7fa4b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:09.884 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 --2- 172.21.15.170:0/694187657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b80f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fa4b4009fb0 tx=0x7fa4b4009b30).stop 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 -- 172.21.15.170:0/694187657 shutdown_connections 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 --2- 172.21.15.170:0/694187657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b80f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 -- 172.21.15.170:0/694187657 >> 172.21.15.170:0/694187657 conn(0x7fa4b80f0b70 msgr2=0x7fa4b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 -- 172.21.15.170:0/694187657 shutdown_connections 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.881+0000 7fa4c06ef700 1 -- 172.21.15.170:0/694187657 wait complete. 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4c06ef700 1 Processor -- start 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4c06ef700 1 -- start start 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4c06ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4c06ef700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b8060280 con 0x7fa4b80f5410 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4be48b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4be48b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:53728/0 (socket says 172.21.15.170:53728) 2023-12-08T10:13:09.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4be48b700 1 -- 172.21.15.170:0/3645714872 learned_addr learned my addr 172.21.15.170:0/3645714872 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.882+0000 7fa4be48b700 1 -- 172.21.15.170:0/3645714872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b4005040 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.883+0000 7fa4be48b700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b81093f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa4b4014000 tx=0x7fa4b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.883+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b40095f0 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.883+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b8060480 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.883+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4b4007e70 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.883+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b4014ea0 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.883+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b8109b20 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.884+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4b4025070 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.884+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b8045ad0 con 0x7fa4b80f5410 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.884+0000 7fa4aeffd700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4a402ff60 0x7fa4a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:09.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.884+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4b4040a40 con 0x7fa4b80f5410 2023-12-08T10:13:09.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.884+0000 7fa4bdc8a700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4a402ff60 0x7fa4a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:09.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.884+0000 7fa4bdc8a700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4a402ff60 0x7fa4a4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa4a80096a0 tx=0x7fa4a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:09.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:09.889+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4b40079c0 con 0x7fa4b80f5410 2023-12-08T10:13:10.098 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.096+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4b810a340 con 0x7fa4b80f5410 2023-12-08T10:13:10.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.096+0000 7fa4aeffd700 1 -- 172.21.15.170:0/3645714872 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4b400f300 con 0x7fa4b80f5410 2023-12-08T10:13:10.099 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:10.099 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:10.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4a402ff60 msgr2=0x7fa4a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:10.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4a402ff60 0x7fa4a4032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa4a80096a0 tx=0x7fa4a8006b40).stop 2023-12-08T10:13:10.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 msgr2=0x7fa4b81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:10.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b81093f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fa4b4014000 tx=0x7fa4b400b040).stop 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 shutdown_connections 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fa4a402ff60 0x7fa4a4032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 --2- 172.21.15.170:0/3645714872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4b80f5410 0x7fa4b81093f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 >> 172.21.15.170:0/3645714872 conn(0x7fa4b80f0b70 msgr2=0x7fa4b80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 shutdown_connections 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:10.098+0000 7fa4c06ef700 1 -- 172.21.15.170:0/3645714872 wait complete. 2023-12-08T10:13:10.101 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:08.369595+0000 mgr.smithi023.tgiigg (mgr.14164) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:11.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:10.098304+0000 mon.smithi023 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.170:0/3645714872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:11.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:11.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:12.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:10.369972+0000 mgr.smithi023.tgiigg (mgr.14164) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:12.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 -- 172.21.15.170:0/2960823587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 msgr2=0x7f55640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 --2- 172.21.15.170:0/2960823587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f55640f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f5558009fb0 tx=0x7f5558009b30).stop 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 -- 172.21.15.170:0/2960823587 shutdown_connections 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 --2- 172.21.15.170:0/2960823587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f55640f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 -- 172.21.15.170:0/2960823587 >> 172.21.15.170:0/2960823587 conn(0x7f55640f0b70 msgr2=0x7f55640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 -- 172.21.15.170:0/2960823587 shutdown_connections 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.676+0000 7f556be7c700 1 -- 172.21.15.170:0/2960823587 wait complete. 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f556be7c700 1 Processor -- start 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f556be7c700 1 -- start start 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f556be7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f5564105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f556be7c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5564105b20 con 0x7f55640f5410 2023-12-08T10:13:12.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f5569c18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f5564105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f5569c18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f5564105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59052/0 (socket says 172.21.15.170:59052) 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.677+0000 7f5569c18700 1 -- 172.21.15.170:0/4094345833 learned_addr learned my addr 172.21.15.170:0/4094345833 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f5569c18700 1 -- 172.21.15.170:0/4094345833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558005040 con 0x7f55640f5410 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f5569c18700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f5564105630 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f5558013510 tx=0x7f555800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5558009320 con 0x7f55640f5410 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5564105d20 con 0x7f55640f5410 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5558013f00 con 0x7f55640f5410 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5558014d90 con 0x7f55640f5410 2023-12-08T10:13:12.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.678+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55641061b0 con 0x7f55640f5410 2023-12-08T10:13:12.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.679+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55580095f0 con 0x7f55640f5410 2023-12-08T10:13:12.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.679+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5564045ad0 con 0x7f55640f5410 2023-12-08T10:13:12.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.679+0000 7f55567fc700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f555002ff60 0x7f5550032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:12.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.679+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5558041000 con 0x7f55640f5410 2023-12-08T10:13:12.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.679+0000 7f5569417700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f555002ff60 0x7f5550032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:12.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.680+0000 7f5569417700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f555002ff60 0x7f5550032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f55600096a0 tx=0x7f5560006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:12.686 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.684+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5558014420 con 0x7f55640f5410 2023-12-08T10:13:12.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.889+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55640fae60 con 0x7f55640f5410 2023-12-08T10:13:12.892 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.890+0000 7f55567fc700 1 -- 172.21.15.170:0/4094345833 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f555801d7f0 con 0x7f55640f5410 2023-12-08T10:13:12.892 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:12.892 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:12.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f555002ff60 msgr2=0x7f5550032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:12.893 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f555002ff60 0x7f5550032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f55600096a0 tx=0x7f5560006b40).stop 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 msgr2=0x7f5564105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f5564105630 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f5558013510 tx=0x7f555800b040).stop 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 shutdown_connections 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f555002ff60 0x7f5550032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 --2- 172.21.15.170:0/4094345833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55640f5410 0x7f5564105630 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 >> 172.21.15.170:0/4094345833 conn(0x7f55640f0b70 msgr2=0x7f55640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 shutdown_connections 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:12.891+0000 7f556be7c700 1 -- 172.21.15.170:0/4094345833 wait complete. 2023-12-08T10:13:12.894 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:12.370315+0000 mgr.smithi023.tgiigg (mgr.14164) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:14.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:13 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:12.891351+0000 mon.smithi023 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.170:0/4094345833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:14.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:14.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:15.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 -- 172.21.15.170:0/3759475999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 msgr2=0x7f2c740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:15.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 --2- 172.21.15.170:0/3759475999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c740f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f2c5c009fb0 tx=0x7f2c5c009b30).stop 2023-12-08T10:13:15.395 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 -- 172.21.15.170:0/3759475999 shutdown_connections 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 --2- 172.21.15.170:0/3759475999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c740f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 -- 172.21.15.170:0/3759475999 >> 172.21.15.170:0/3759475999 conn(0x7f2c740f0b70 msgr2=0x7f2c740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 -- 172.21.15.170:0/3759475999 shutdown_connections 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.392+0000 7f2c79c27700 1 -- 172.21.15.170:0/3759475999 wait complete. 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c79c27700 1 Processor -- start 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c79c27700 1 -- start start 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c79c27700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c74106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:15.396 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c79c27700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c74106c50 con 0x7f2c740f5410 2023-12-08T10:13:15.397 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c74106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c74106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59076/0 (socket says 172.21.15.170:59076) 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.393+0000 7f2c737fe700 1 -- 172.21.15.170:0/375400725 learned_addr learned my addr 172.21.15.170:0/375400725 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c737fe700 1 -- 172.21.15.170:0/375400725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c005040 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c737fe700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c74106760 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f2c5c009f80 tx=0x7f2c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c5c009320 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c74106e50 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c5c013ec0 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c5c01ece0 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.394+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c741072e0 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.395+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c5c014420 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.395+0000 7f2c6bfff700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2c6002ff60 0x7f2c60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.395+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c5c040e30 con 0x7f2c740f5410 2023-12-08T10:13:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.395+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c74045ad0 con 0x7f2c740f5410 2023-12-08T10:13:15.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.395+0000 7f2c72ffd700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2c6002ff60 0x7f2c60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:15.399 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.396+0000 7f2c72ffd700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2c6002ff60 0x7f2c60032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2c640096a0 tx=0x7f2c64006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:15.402 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.400+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c5c01c910 con 0x7f2c740f5410 2023-12-08T10:13:15.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.606+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c74108020 con 0x7f2c740f5410 2023-12-08T10:13:15.609 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c6bfff700 1 -- 172.21.15.170:0/375400725 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c5c01c320 con 0x7f2c740f5410 2023-12-08T10:13:15.610 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:15.610 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:15.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2c6002ff60 msgr2=0x7f2c60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:15.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2c6002ff60 0x7f2c60032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2c640096a0 tx=0x7f2c64006b40).stop 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 msgr2=0x7f2c74106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c74106760 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f2c5c009f80 tx=0x7f2c5c00b040).stop 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 shutdown_connections 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2c6002ff60 0x7f2c60032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 --2- 172.21.15.170:0/375400725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c740f5410 0x7f2c74106760 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 >> 172.21.15.170:0/375400725 conn(0x7f2c740f0b70 msgr2=0x7f2c7405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:15.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 shutdown_connections 2023-12-08T10:13:15.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:15.608+0000 7f2c79c27700 1 -- 172.21.15.170:0/375400725 wait complete. 2023-12-08T10:13:15.613 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:14.370660+0000 mgr.smithi023.tgiigg (mgr.14164) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:16.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:15 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:15.608621+0000 mon.smithi023 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.170:0/375400725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:16.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:16.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:18.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 -- 172.21.15.170:0/2485350580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 msgr2=0x7ff8900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:18.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 --2- 172.21.15.170:0/2485350580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f90f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7ff884009fb0 tx=0x7ff884009b30).stop 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 -- 172.21.15.170:0/2485350580 shutdown_connections 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 --2- 172.21.15.170:0/2485350580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f90f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 -- 172.21.15.170:0/2485350580 >> 172.21.15.170:0/2485350580 conn(0x7ff8900f0b70 msgr2=0x7ff8900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 -- 172.21.15.170:0/2485350580 shutdown_connections 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 -- 172.21.15.170:0/2485350580 wait complete. 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.100+0000 7ff8949ad700 1 Processor -- start 2023-12-08T10:13:18.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff8949ad700 1 -- start start 2023-12-08T10:13:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff8949ad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff8949ad700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8900f4ff0 con 0x7ff890060280 2023-12-08T10:13:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff88e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff88e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59104/0 (socket says 172.21.15.170:59104) 2023-12-08T10:13:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff88e59c700 1 -- 172.21.15.170:0/1029236191 learned_addr learned my addr 172.21.15.170:0/1029236191 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff88e59c700 1 -- 172.21.15.170:0/1029236191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884005040 con 0x7ff890060280 2023-12-08T10:13:18.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff88e59c700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f6b70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7ff884013510 tx=0x7ff88400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:18.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff884009320 con 0x7ff890060280 2023-12-08T10:13:18.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8900f5190 con 0x7ff890060280 2023-12-08T10:13:18.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff884013f00 con 0x7ff890060280 2023-12-08T10:13:18.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.101+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff884014dc0 con 0x7ff890060280 2023-12-08T10:13:18.107 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.102+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8900f5620 con 0x7ff890060280 2023-12-08T10:13:18.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.102+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8840095f0 con 0x7ff890060280 2023-12-08T10:13:18.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.102+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890059100 con 0x7ff890060280 2023-12-08T10:13:18.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.103+0000 7ff87effd700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff87802ff60 0x7ff878032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:18.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.103+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff884041040 con 0x7ff890060280 2023-12-08T10:13:18.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.103+0000 7ff88dd9b700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff87802ff60 0x7ff878032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:18.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.103+0000 7ff88dd9b700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff87802ff60 0x7ff878032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff8800096a0 tx=0x7ff880006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:18.110 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.107+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff884014420 con 0x7ff890060280 2023-12-08T10:13:18.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:16.370950+0000 mgr.smithi023.tgiigg (mgr.14164) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:18.315 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.313+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8900fbf50 con 0x7ff890060280 2023-12-08T10:13:18.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.315+0000 7ff87effd700 1 -- 172.21.15.170:0/1029236191 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff884010230 con 0x7ff890060280 2023-12-08T10:13:18.316 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:18.316 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.315+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff87802ff60 msgr2=0x7ff878032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.315+0000 7ff8949ad700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff87802ff60 0x7ff878032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7ff8800096a0 tx=0x7ff880006b40).stop 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 msgr2=0x7ff8900f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f6b70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7ff884013510 tx=0x7ff88400b040).stop 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 shutdown_connections 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff87802ff60 0x7ff878032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 --2- 172.21.15.170:0/1029236191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff890060280 0x7ff8900f6b70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 >> 172.21.15.170:0/1029236191 conn(0x7ff8900f0b70 msgr2=0x7ff8900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 shutdown_connections 2023-12-08T10:13:18.318 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:18.316+0000 7ff8949ad700 1 -- 172.21.15.170:0/1029236191 wait complete. 2023-12-08T10:13:18.319 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:19.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:18.315696+0000 mon.smithi023 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.170:0/1029236191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:19.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:19.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:20.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:18.371303+0000 mgr.smithi023.tgiigg (mgr.14164) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:20.841 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 -- 172.21.15.170:0/1749062445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 msgr2=0x7fb4280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 --2- 172.21.15.170:0/1749062445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb4280f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fb41c009fb0 tx=0x7fb41c009b30).stop 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 -- 172.21.15.170:0/1749062445 shutdown_connections 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 --2- 172.21.15.170:0/1749062445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb4280f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 -- 172.21.15.170:0/1749062445 >> 172.21.15.170:0/1749062445 conn(0x7fb4280f0b70 msgr2=0x7fb4280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 -- 172.21.15.170:0/1749062445 shutdown_connections 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.838+0000 7fb42c811700 1 -- 172.21.15.170:0/1749062445 wait complete. 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42c811700 1 Processor -- start 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42c811700 1 -- start start 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42c811700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb428109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42c811700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb428060240 con 0x7fb4280f5410 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb428109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:20.842 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb428109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59120/0 (socket says 172.21.15.170:59120) 2023-12-08T10:13:20.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.839+0000 7fb42659c700 1 -- 172.21.15.170:0/613584318 learned_addr learned my addr 172.21.15.170:0/613584318 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:20.843 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb42659c700 1 -- 172.21.15.170:0/613584318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c005040 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb42659c700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb428109430 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb41c013510 tx=0x7fb41c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb41c009320 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb428060440 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb41c013f00 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb41c014d90 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.840+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb428109ae0 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.841+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb41c0263a0 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.841+0000 7fb416ffd700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb41002ff60 0x7fb410032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.841+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb428045ad0 con 0x7fb4280f5410 2023-12-08T10:13:20.844 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.841+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb41c041000 con 0x7fb4280f5410 2023-12-08T10:13:20.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.841+0000 7fb425d9b700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb41002ff60 0x7fb410032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:20.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.842+0000 7fb425d9b700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb41002ff60 0x7fb410032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb4180096a0 tx=0x7fb418006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:20.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:20.847+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb41c06d2b0 con 0x7fb4280f5410 2023-12-08T10:13:21.052 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.050+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4280faf50 con 0x7fb4280f5410 2023-12-08T10:13:21.053 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb416ffd700 1 -- 172.21.15.170:0/613584318 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb41c0095f0 con 0x7fb4280f5410 2023-12-08T10:13:21.053 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:21.053 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:21.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb41002ff60 msgr2=0x7fb410032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:21.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb41002ff60 0x7fb410032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb4180096a0 tx=0x7fb418006b40).stop 2023-12-08T10:13:21.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 msgr2=0x7fb428109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:21.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb428109430 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fb41c013510 tx=0x7fb41c00b040).stop 2023-12-08T10:13:21.055 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 shutdown_connections 2023-12-08T10:13:21.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb41002ff60 0x7fb410032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:21.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 --2- 172.21.15.170:0/613584318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4280f5410 0x7fb428109430 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:21.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.052+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 >> 172.21.15.170:0/613584318 conn(0x7fb4280f0b70 msgr2=0x7fb4280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:21.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.053+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 shutdown_connections 2023-12-08T10:13:21.056 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:21.053+0000 7fb42c811700 1 -- 172.21.15.170:0/613584318 wait complete. 2023-12-08T10:13:21.056 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:20.371583+0000 mgr.smithi023.tgiigg (mgr.14164) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:22.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:21 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:21.052675+0000 mon.smithi023 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.170:0/613584318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:22.427 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:22.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:22.455851+0000 mon.smithi023 (mon.0) 280 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:13:23.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:22.457350+0000 mon.smithi023 (mon.0) 281 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:13:23.576 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- 172.21.15.170:0/3655012746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 msgr2=0x7f4e280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 --2- 172.21.15.170:0/3655012746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e280f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4e24009fb0 tx=0x7f4e24009b30).stop 2023-12-08T10:13:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- 172.21.15.170:0/3655012746 shutdown_connections 2023-12-08T10:13:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 --2- 172.21.15.170:0/3655012746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e280f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- 172.21.15.170:0/3655012746 >> 172.21.15.170:0/3655012746 conn(0x7f4e280f09a0 msgr2=0x7f4e280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- 172.21.15.170:0/3655012746 shutdown_connections 2023-12-08T10:13:23.577 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- 172.21.15.170:0/3655012746 wait complete. 2023-12-08T10:13:23.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 Processor -- start 2023-12-08T10:13:23.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- start start 2023-12-08T10:13:23.578 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e28109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:23.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.572+0000 7f4e2e536700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e28060280 con 0x7f4e280f51a0 2023-12-08T10:13:23.579 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e2d534700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e28109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e2d534700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e28109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36620/0 (socket says 172.21.15.170:36620) 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e2d534700 1 -- 172.21.15.170:0/2518022997 learned_addr learned my addr 172.21.15.170:0/2518022997 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e2d534700 1 -- 172.21.15.170:0/2518022997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24005040 con 0x7f4e280f51a0 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e2d534700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e28109170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4e24013510 tx=0x7f4e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e24009320 con 0x7f4e280f51a0 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e28060480 con 0x7f4e280f51a0 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e24013f00 con 0x7f4e280f51a0 2023-12-08T10:13:23.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.573+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e24014dc0 con 0x7f4e280f51a0 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.574+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e281098a0 con 0x7f4e280f51a0 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.574+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e240095f0 con 0x7f4e280f51a0 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.574+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e28045ad0 con 0x7f4e280f51a0 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.574+0000 7f4e1dffb700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e1402ff60 0x7f4e14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.575+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e24041020 con 0x7f4e280f51a0 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.575+0000 7f4e2cd33700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e1402ff60 0x7f4e14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:23.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.575+0000 7f4e2cd33700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e1402ff60 0x7f4e14032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4e180096a0 tx=0x7f4e18006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:23.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.579+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e24014420 con 0x7f4e280f51a0 2023-12-08T10:13:23.787 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.786+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e280fabe0 con 0x7f4e280f51a0 2023-12-08T10:13:23.788 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.787+0000 7f4e1dffb700 1 -- 172.21.15.170:0/2518022997 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4e2401c920 con 0x7f4e280f51a0 2023-12-08T10:13:23.788 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:23.788 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e1402ff60 msgr2=0x7f4e14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e1402ff60 0x7f4e14032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4e180096a0 tx=0x7f4e18006b40).stop 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 msgr2=0x7f4e28109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e28109170 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f4e24013510 tx=0x7f4e2400b040).stop 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 shutdown_connections 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4e1402ff60 0x7f4e14032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 --2- 172.21.15.170:0/2518022997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e280f51a0 0x7f4e28109170 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:23.790 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 >> 172.21.15.170:0/2518022997 conn(0x7f4e280f09a0 msgr2=0x7f4e280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:23.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.788+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 shutdown_connections 2023-12-08T10:13:23.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:23.789+0000 7f4e2e536700 1 -- 172.21.15.170:0/2518022997 wait complete. 2023-12-08T10:13:23.791 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:22.371925+0000 mgr.smithi023.tgiigg (mgr.14164) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:24.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:23.787502+0000 mon.smithi023 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.170:0/2518022997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:25.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:25.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:26.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:25 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:24.372320+0000 mgr.smithi023.tgiigg (mgr.14164) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:26.273 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.271+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/2474250363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 msgr2=0x7fb5a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.271+0000 7fb5ae5e5700 1 --2- 172.21.15.170:0/2474250363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a80f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fb598009fb0 tx=0x7fb598009b30).stop 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/2474250363 shutdown_connections 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 --2- 172.21.15.170:0/2474250363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a80f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/2474250363 >> 172.21.15.170:0/2474250363 conn(0x7fb5a80f0b70 msgr2=0x7fb5a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/2474250363 shutdown_connections 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/2474250363 wait complete. 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 Processor -- start 2023-12-08T10:13:26.274 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.272+0000 7fb5ae5e5700 1 -- start start 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.273+0000 7fb5ae5e5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.273+0000 7fb5ae5e5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8105b20 con 0x7fb5a80f5410 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.273+0000 7fb5a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.273+0000 7fb5a7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36650/0 (socket says 172.21.15.170:36650) 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.273+0000 7fb5a7fff700 1 -- 172.21.15.170:0/937496163 learned_addr learned my addr 172.21.15.170:0/937496163 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.273+0000 7fb5a7fff700 1 -- 172.21.15.170:0/937496163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598005040 con 0x7fb5a80f5410 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.274+0000 7fb5a7fff700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a8105630 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb598013510 tx=0x7fb59800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:26.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.274+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb598009320 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.274+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a8105d20 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.274+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb598013f00 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.274+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb598014d90 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.274+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a81061b0 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.275+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb598014420 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.275+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a8045ad0 con 0x7fb5a80f5410 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.275+0000 7fb5a4ff9700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb59002ff60 0x7fb590032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:26.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.275+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb598042df0 con 0x7fb5a80f5410 2023-12-08T10:13:26.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.275+0000 7fb5a77fe700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb59002ff60 0x7fb590032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:26.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.276+0000 7fb5a77fe700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb59002ff60 0x7fb590032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb59c0096a0 tx=0x7fb59c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:26.281 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.280+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5980095f0 con 0x7fb5a80f5410 2023-12-08T10:13:26.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.487+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5a80fae60 con 0x7fb5a80f5410 2023-12-08T10:13:26.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.488+0000 7fb5a4ff9700 1 -- 172.21.15.170:0/937496163 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb59801fb70 con 0x7fb5a80f5410 2023-12-08T10:13:26.490 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:26.490 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.489+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb59002ff60 msgr2=0x7fb590032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.489+0000 7fb5ae5e5700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb59002ff60 0x7fb590032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fb59c0096a0 tx=0x7fb59c006b40).stop 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 msgr2=0x7fb5a8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a8105630 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb598013510 tx=0x7fb59800b040).stop 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 shutdown_connections 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb59002ff60 0x7fb590032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 --2- 172.21.15.170:0/937496163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5a80f5410 0x7fb5a8105630 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 >> 172.21.15.170:0/937496163 conn(0x7fb5a80f0b70 msgr2=0x7fb5a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:26.491 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 shutdown_connections 2023-12-08T10:13:26.492 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:26.490+0000 7fb5ae5e5700 1 -- 172.21.15.170:0/937496163 wait complete. 2023-12-08T10:13:26.492 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:27.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:26 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:26.488728+0000 mon.smithi023 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.170:0/937496163' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:27.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:27.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:28.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:26.372668+0000 mgr.smithi023.tgiigg (mgr.14164) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 -- 172.21.15.170:0/2148157672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 msgr2=0x7f63000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 --2- 172.21.15.170:0/2148157672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f63000f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f62f4009fb0 tx=0x7f62f4009b30).stop 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 -- 172.21.15.170:0/2148157672 shutdown_connections 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 --2- 172.21.15.170:0/2148157672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f63000f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 -- 172.21.15.170:0/2148157672 >> 172.21.15.170:0/2148157672 conn(0x7f63000f1c70 msgr2=0x7f63000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 -- 172.21.15.170:0/2148157672 shutdown_connections 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.039+0000 7f6306f30700 1 -- 172.21.15.170:0/2148157672 wait complete. 2023-12-08T10:13:29.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.040+0000 7f6306f30700 1 Processor -- start 2023-12-08T10:13:29.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.040+0000 7f6306f30700 1 -- start start 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.040+0000 7f6306f30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f6300106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.040+0000 7f6306f30700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6300106c20 con 0x7f63000f6510 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6304ccc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f6300106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6304ccc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f6300106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36670/0 (socket says 172.21.15.170:36670) 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6304ccc700 1 -- 172.21.15.170:0/2664153866 learned_addr learned my addr 172.21.15.170:0/2664153866 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6304ccc700 1 -- 172.21.15.170:0/2664153866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4005040 con 0x7f63000f6510 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6304ccc700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f6300106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f62f4013510 tx=0x7f62f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62f4009320 con 0x7f63000f6510 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6300106e20 con 0x7f63000f6510 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63001072b0 con 0x7f63000f6510 2023-12-08T10:13:29.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62f4013f00 con 0x7f63000f6510 2023-12-08T10:13:29.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.041+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62f4014d60 con 0x7f63000f6510 2023-12-08T10:13:29.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.042+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62f4014420 con 0x7f63000f6510 2023-12-08T10:13:29.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.042+0000 7f62fd7fa700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f62f002ffb0 0x7f62f0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:29.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.042+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62f4041390 con 0x7f63000f6510 2023-12-08T10:13:29.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.042+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6300059100 con 0x7f63000f6510 2023-12-08T10:13:29.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.042+0000 7f62fffff700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f62f002ffb0 0x7f62f0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:29.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.043+0000 7f62fffff700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f62f002ffb0 0x7f62f0032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f62ec0096a0 tx=0x7f62ec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:29.048 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.047+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62f4012b20 con 0x7f63000f6510 2023-12-08T10:13:29.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.252+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63000fbf60 con 0x7f63000f6510 2023-12-08T10:13:29.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.253+0000 7f62fd7fa700 1 -- 172.21.15.170:0/2664153866 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62f402b070 con 0x7f63000f6510 2023-12-08T10:13:29.254 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:29.255 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.254+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f62f002ffb0 msgr2=0x7f62f0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.254+0000 7f6306f30700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f62f002ffb0 0x7f62f0032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f62ec0096a0 tx=0x7f62ec006b40).stop 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.254+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 msgr2=0x7f6300106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.254+0000 7f6306f30700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f6300106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f62f4013510 tx=0x7f62f400b040).stop 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.255+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 shutdown_connections 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.255+0000 7f6306f30700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f62f002ffb0 0x7f62f0032470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.255+0000 7f6306f30700 1 --2- 172.21.15.170:0/2664153866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63000f6510 0x7f6300106730 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.255+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 >> 172.21.15.170:0/2664153866 conn(0x7f63000f1c70 msgr2=0x7f63000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.255+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 shutdown_connections 2023-12-08T10:13:29.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:29.255+0000 7f6306f30700 1 -- 172.21.15.170:0/2664153866 wait complete. 2023-12-08T10:13:29.257 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:28.373006+0000 mgr.smithi023.tgiigg (mgr.14164) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:30.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:29 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:29.253701+0000 mon.smithi023 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.170:0/2664153866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:30.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:30.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 -- 172.21.15.170:0/4248638192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 msgr2=0x7fd4500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 --2- 172.21.15.170:0/4248638192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd4500f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fd444009fb0 tx=0x7fd444009b30).stop 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 -- 172.21.15.170:0/4248638192 shutdown_connections 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 --2- 172.21.15.170:0/4248638192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd4500f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 -- 172.21.15.170:0/4248638192 >> 172.21.15.170:0/4248638192 conn(0x7fd4500f0b70 msgr2=0x7fd4500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 -- 172.21.15.170:0/4248638192 shutdown_connections 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.787+0000 7fd456fe0700 1 -- 172.21.15.170:0/4248638192 wait complete. 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd456fe0700 1 Processor -- start 2023-12-08T10:13:31.789 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd456fe0700 1 -- start start 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd456fe0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd450105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd456fe0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd450105b20 con 0x7fd4500f5410 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd454d7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd450105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd454d7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd450105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:36694/0 (socket says 172.21.15.170:36694) 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.788+0000 7fd454d7c700 1 -- 172.21.15.170:0/4008489300 learned_addr learned my addr 172.21.15.170:0/4008489300 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd454d7c700 1 -- 172.21.15.170:0/4008489300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd444005040 con 0x7fd4500f5410 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd454d7c700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd450105630 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fd444013510 tx=0x7fd44400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd444009320 con 0x7fd4500f5410 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd450105d20 con 0x7fd4500f5410 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd444013f00 con 0x7fd4500f5410 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd444014d90 con 0x7fd4500f5410 2023-12-08T10:13:31.791 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.789+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4501061b0 con 0x7fd4500f5410 2023-12-08T10:13:31.792 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.790+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4440095f0 con 0x7fd4500f5410 2023-12-08T10:13:31.792 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.790+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd450045ad0 con 0x7fd4500f5410 2023-12-08T10:13:31.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.790+0000 7fd44d7fa700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd44002ff60 0x7fd440032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:31.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.790+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd444041000 con 0x7fd4500f5410 2023-12-08T10:13:31.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.790+0000 7fd44ffff700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd44002ff60 0x7fd440032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:31.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.791+0000 7fd44ffff700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd44002ff60 0x7fd440032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd43c0096a0 tx=0x7fd43c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:31.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:31.795+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd444014420 con 0x7fd4500f5410 2023-12-08T10:13:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.001+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4500fae60 con 0x7fd4500f5410 2023-12-08T10:13:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.002+0000 7fd44d7fa700 1 -- 172.21.15.170:0/4008489300 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd444010230 con 0x7fd4500f5410 2023-12-08T10:13:32.004 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:32.004 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.003+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd44002ff60 msgr2=0x7fd440032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.003+0000 7fd456fe0700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd44002ff60 0x7fd440032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd43c0096a0 tx=0x7fd43c006b40).stop 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 msgr2=0x7fd450105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd450105630 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fd444013510 tx=0x7fd44400b040).stop 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 shutdown_connections 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd44002ff60 0x7fd440032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 --2- 172.21.15.170:0/4008489300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4500f5410 0x7fd450105630 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:32.005 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 >> 172.21.15.170:0/4008489300 conn(0x7fd4500f0b70 msgr2=0x7fd4500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:32.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 shutdown_connections 2023-12-08T10:13:32.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:32.004+0000 7fd456fe0700 1 -- 172.21.15.170:0/4008489300 wait complete. 2023-12-08T10:13:32.006 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:32.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:31 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:30.373386+0000 mgr.smithi023.tgiigg (mgr.14164) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:33.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:32.002897+0000 mon.smithi023 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.170:0/4008489300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:33.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:33.402 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:34.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:32.373680+0000 mgr.smithi023.tgiigg (mgr.14164) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3420899437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 msgr2=0x7f2b0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 --2- 172.21.15.170:0/3420899437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f2b00009fb0 tx=0x7f2b00009b30).stop 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3420899437 shutdown_connections 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 --2- 172.21.15.170:0/3420899437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3420899437 >> 172.21.15.170:0/3420899437 conn(0x7f2b0c0f1c70 msgr2=0x7f2b0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3420899437 shutdown_connections 2023-12-08T10:13:34.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.556+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3420899437 wait complete. 2023-12-08T10:13:34.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b12e3a700 1 Processor -- start 2023-12-08T10:13:34.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b12e3a700 1 -- start start 2023-12-08T10:13:34.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b12e3a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:34.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b12e3a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0c108e20 con 0x7f2b0c0f6510 2023-12-08T10:13:34.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b10bd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:34.560 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b10bd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33790/0 (socket says 172.21.15.170:33790) 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.557+0000 7f2b10bd6700 1 -- 172.21.15.170:0/3726183823 learned_addr learned my addr 172.21.15.170:0/3726183823 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b10bd6700 1 -- 172.21.15.170:0/3726183823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00005040 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b10bd6700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c108930 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f2b00013510 tx=0x7f2b0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b00009320 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0c109020 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b00013f00 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b00014d90 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.558+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b0c1094b0 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.559+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b000263a0 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.559+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0c045ad0 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.559+0000 7f2b097fa700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2afc02ff60 0x7f2afc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.559+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b000412d0 con 0x7f2b0c0f6510 2023-12-08T10:13:34.561 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.559+0000 7f2b0bfff700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2afc02ff60 0x7f2afc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:34.562 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.560+0000 7f2b0bfff700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2afc02ff60 0x7f2afc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f2af80096a0 tx=0x7f2af8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:34.565 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.564+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b0001cb10 con 0x7f2b0c0f6510 2023-12-08T10:13:34.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.767+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b0c0fbf60 con 0x7f2b0c0f6510 2023-12-08T10:13:34.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.768+0000 7f2b097fa700 1 -- 172.21.15.170:0/3726183823 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b0001cb10 con 0x7f2b0c0f6510 2023-12-08T10:13:34.770 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:34.770 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2afc02ff60 msgr2=0x7f2afc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2afc02ff60 0x7f2afc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f2af80096a0 tx=0x7f2af8006b40).stop 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 msgr2=0x7f2b0c108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c108930 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f2b00013510 tx=0x7f2b0000b040).stop 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 shutdown_connections 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f2afc02ff60 0x7f2afc032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 --2- 172.21.15.170:0/3726183823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b0c0f6510 0x7f2b0c108930 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 >> 172.21.15.170:0/3726183823 conn(0x7f2b0c0f1c70 msgr2=0x7f2b0c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:34.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 shutdown_connections 2023-12-08T10:13:34.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:34.770+0000 7f2b12e3a700 1 -- 172.21.15.170:0/3726183823 wait complete. 2023-12-08T10:13:34.772 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:35.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:34 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:34.768889+0000 mon.smithi023 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.170:0/3726183823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:36.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:36.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:36.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:34.374020+0000 mgr.smithi023.tgiigg (mgr.14164) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:37.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 -- 172.21.15.170:0/750053078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 msgr2=0x7f5c580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:37.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 --2- 172.21.15.170:0/750053078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c580f90f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f5c4c009fb0 tx=0x7f5c4c009b30).stop 2023-12-08T10:13:37.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 -- 172.21.15.170:0/750053078 shutdown_connections 2023-12-08T10:13:37.333 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 --2- 172.21.15.170:0/750053078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c580f90f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 -- 172.21.15.170:0/750053078 >> 172.21.15.170:0/750053078 conn(0x7f5c580f0b70 msgr2=0x7f5c580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 -- 172.21.15.170:0/750053078 shutdown_connections 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.331+0000 7f5c5dba9700 1 -- 172.21.15.170:0/750053078 wait complete. 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c5dba9700 1 Processor -- start 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c5dba9700 1 -- start start 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c5dba9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c581066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c5dba9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c58106bd0 con 0x7f5c58060280 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c577fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c581066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c577fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c581066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33810/0 (socket says 172.21.15.170:33810) 2023-12-08T10:13:37.334 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c577fe700 1 -- 172.21.15.170:0/2382661387 learned_addr learned my addr 172.21.15.170:0/2382661387 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.332+0000 7f5c577fe700 1 -- 172.21.15.170:0/2382661387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c005040 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.333+0000 7f5c577fe700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c581066e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5c4c013510 tx=0x7f5c4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.333+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4c009320 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.333+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c58106dd0 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.333+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c4c013f00 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.333+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c4c014dc0 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.333+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c58107260 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.334+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c4c014420 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.334+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c58045ad0 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.334+0000 7f5c3ffff700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c4002ff60 0x7f5c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.334+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c4c0410f0 con 0x7f5c58060280 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.334+0000 7f5c56ffd700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c4002ff60 0x7f5c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:37.336 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.334+0000 7f5c56ffd700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c4002ff60 0x7f5c40032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5c480096a0 tx=0x7f5c48006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:37.340 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.339+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c4c0095f0 con 0x7f5c58060280 2023-12-08T10:13:37.546 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.545+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c58108110 con 0x7f5c58060280 2023-12-08T10:13:37.547 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.546+0000 7f5c3ffff700 1 -- 172.21.15.170:0/2382661387 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c4c01cb10 con 0x7f5c58060280 2023-12-08T10:13:37.547 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:37.547 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c4002ff60 msgr2=0x7f5c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c4002ff60 0x7f5c40032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f5c480096a0 tx=0x7f5c48006b40).stop 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 msgr2=0x7f5c581066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c581066e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5c4c013510 tx=0x7f5c4c00b040).stop 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 shutdown_connections 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f5c4002ff60 0x7f5c40032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 --2- 172.21.15.170:0/2382661387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5c58060280 0x7f5c581066e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 >> 172.21.15.170:0/2382661387 conn(0x7f5c580f0b70 msgr2=0x7f5c580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 shutdown_connections 2023-12-08T10:13:37.549 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:37.548+0000 7f5c5dba9700 1 -- 172.21.15.170:0/2382661387 wait complete. 2023-12-08T10:13:37.550 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:38.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:36.374408+0000 mgr.smithi023.tgiigg (mgr.14164) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:38.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:37.546705+0000 mon.smithi023 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.170:0/2382661387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:38.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:38.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:40.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 -- 172.21.15.170:0/1567246547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 msgr2=0x7f11a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:40.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 --2- 172.21.15.170:0/1567246547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a80f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f1198009fb0 tx=0x7f1198009b30).stop 2023-12-08T10:13:40.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 -- 172.21.15.170:0/1567246547 shutdown_connections 2023-12-08T10:13:40.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 --2- 172.21.15.170:0/1567246547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a80f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:40.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 -- 172.21.15.170:0/1567246547 >> 172.21.15.170:0/1567246547 conn(0x7f11a80f09a0 msgr2=0x7f11a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:40.075 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 -- 172.21.15.170:0/1567246547 shutdown_connections 2023-12-08T10:13:40.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 -- 172.21.15.170:0/1567246547 wait complete. 2023-12-08T10:13:40.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 Processor -- start 2023-12-08T10:13:40.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.070+0000 7f11aea2f700 1 -- start start 2023-12-08T10:13:40.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11aea2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:40.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11aea2f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a8060220 con 0x7f11a80f51a0 2023-12-08T10:13:40.076 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11ada2d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11ada2d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:33832/0 (socket says 172.21.15.170:33832) 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11ada2d700 1 -- 172.21.15.170:0/3278163379 learned_addr learned my addr 172.21.15.170:0/3278163379 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11ada2d700 1 -- 172.21.15.170:0/3278163379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1198005040 con 0x7f11a80f51a0 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11ada2d700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a81091d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1198009830 tx=0x7f119800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1198009320 con 0x7f11a80f51a0 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a8060420 con 0x7f11a80f51a0 2023-12-08T10:13:40.077 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11980136c0 con 0x7f11a80f51a0 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f119801ee90 con 0x7f11a80f51a0 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.071+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a8109880 con 0x7f11a80f51a0 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.072+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1198014440 con 0x7f11a80f51a0 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.072+0000 7f119e7fc700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f119402ff10 0x7f11940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.072+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1198040ee0 con 0x7f11a80f51a0 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.072+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a8059100 con 0x7f11a80f51a0 2023-12-08T10:13:40.078 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.072+0000 7f11ad22c700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f119402ff10 0x7f11940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:40.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.073+0000 7f11ad22c700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f119402ff10 0x7f11940323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f11a40096a0 tx=0x7f11a4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:40.079 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.077+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11980273f0 con 0x7f11a80f51a0 2023-12-08T10:13:40.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:38.374725+0000 mgr.smithi023.tgiigg (mgr.14164) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:40.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.285+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11a8109f70 con 0x7f11a80f51a0 2023-12-08T10:13:40.287 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.286+0000 7f119e7fc700 1 -- 172.21.15.170:0/3278163379 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f119801c320 con 0x7f11a80f51a0 2023-12-08T10:13:40.287 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:40.287 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f119402ff10 msgr2=0x7f11940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f119402ff10 0x7f11940323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f11a40096a0 tx=0x7f11a4006b40).stop 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 msgr2=0x7f11a81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a81091d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f1198009830 tx=0x7f119800b040).stop 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 shutdown_connections 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f119402ff10 0x7f11940323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 --2- 172.21.15.170:0/3278163379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11a80f51a0 0x7f11a81091d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.287+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 >> 172.21.15.170:0/3278163379 conn(0x7f11a80f09a0 msgr2=0x7f11a80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.288+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 shutdown_connections 2023-12-08T10:13:40.289 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:40.288+0000 7f11aea2f700 1 -- 172.21.15.170:0/3278163379 wait complete. 2023-12-08T10:13:40.290 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:41.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:40.286732+0000 mon.smithi023 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.170:0/3278163379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:41.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:41.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:42.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:40.375029+0000 mgr.smithi023.tgiigg (mgr.14164) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 -- 172.21.15.170:0/333294167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 msgr2=0x7f81080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 --2- 172.21.15.170:0/333294167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f81080f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f80f8009fb0 tx=0x7f80f8009b30).stop 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 -- 172.21.15.170:0/333294167 shutdown_connections 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 --2- 172.21.15.170:0/333294167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f81080f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 -- 172.21.15.170:0/333294167 >> 172.21.15.170:0/333294167 conn(0x7f81080f1c70 msgr2=0x7f81080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 -- 172.21.15.170:0/333294167 shutdown_connections 2023-12-08T10:13:42.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.769+0000 7f811008d700 1 -- 172.21.15.170:0/333294167 wait complete. 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f811008d700 1 Processor -- start 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f811008d700 1 -- start start 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f811008d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f810806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f811008d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108069e20 con 0x7f81080f6510 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f810de29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f810806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f810de29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f810806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59058/0 (socket says 172.21.15.170:59058) 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.770+0000 7f810de29700 1 -- 172.21.15.170:0/1329058360 learned_addr learned my addr 172.21.15.170:0/1329058360 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f810de29700 1 -- 172.21.15.170:0/1329058360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f8005040 con 0x7f81080f6510 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f810de29700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f810806b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f80f8013510 tx=0x7f80f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f8009320 con 0x7f81080f6510 2023-12-08T10:13:42.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8108069fc0 con 0x7f81080f6510 2023-12-08T10:13:42.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810806a450 con 0x7f81080f6510 2023-12-08T10:13:42.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80f8013f00 con 0x7f81080f6510 2023-12-08T10:13:42.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.771+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f8014d60 con 0x7f81080f6510 2023-12-08T10:13:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.772+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80f80263a0 con 0x7f81080f6510 2023-12-08T10:13:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.772+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8108059100 con 0x7f81080f6510 2023-12-08T10:13:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.772+0000 7f80fe7fc700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80f402ff60 0x7f80f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.772+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80f80412b0 con 0x7f81080f6510 2023-12-08T10:13:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.772+0000 7f810d628700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80f402ff60 0x7f80f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:42.775 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.773+0000 7f810d628700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80f402ff60 0x7f80f4032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f81040096a0 tx=0x7f8104006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:42.778 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.777+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80f801c320 con 0x7f81080f6510 2023-12-08T10:13:42.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.983+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81080fbf60 con 0x7f81080f6510 2023-12-08T10:13:42.986 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.984+0000 7f80fe7fc700 1 -- 172.21.15.170:0/1329058360 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80f800fe60 con 0x7f81080f6510 2023-12-08T10:13:42.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:42.986 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:42.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80f402ff60 msgr2=0x7f80f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:42.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80f402ff60 0x7f80f4032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f81040096a0 tx=0x7f8104006b40).stop 2023-12-08T10:13:42.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 msgr2=0x7f810806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:42.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f810806b9a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f80f8013510 tx=0x7f80f800b040).stop 2023-12-08T10:13:42.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 shutdown_connections 2023-12-08T10:13:42.990 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80f402ff60 0x7f80f4032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:42.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 --2- 172.21.15.170:0/1329058360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f81080f6510 0x7f810806b9a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:42.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 >> 172.21.15.170:0/1329058360 conn(0x7f81080f1c70 msgr2=0x7f81080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:42.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 shutdown_connections 2023-12-08T10:13:42.991 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:42.985+0000 7f811008d700 1 -- 172.21.15.170:0/1329058360 wait complete. 2023-12-08T10:13:42.991 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:44.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:44.365 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:44.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:42.375384+0000 mgr.smithi023.tgiigg (mgr.14164) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:44.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:42.985495+0000 mon.smithi023 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.170:0/1329058360' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.512+0000 7fad25da1700 1 -- 172.21.15.170:0/299020213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 msgr2=0x7fad200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.512+0000 7fad25da1700 1 --2- 172.21.15.170:0/299020213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad200f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fad14009fb0 tx=0x7fad14009b30).stop 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.512+0000 7fad25da1700 1 -- 172.21.15.170:0/299020213 shutdown_connections 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.512+0000 7fad25da1700 1 --2- 172.21.15.170:0/299020213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad200f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.512+0000 7fad25da1700 1 -- 172.21.15.170:0/299020213 >> 172.21.15.170:0/299020213 conn(0x7fad200f0b70 msgr2=0x7fad200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.513+0000 7fad25da1700 1 -- 172.21.15.170:0/299020213 shutdown_connections 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.513+0000 7fad25da1700 1 -- 172.21.15.170:0/299020213 wait complete. 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.513+0000 7fad25da1700 1 Processor -- start 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.513+0000 7fad25da1700 1 -- start start 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.513+0000 7fad25da1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad201088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:45.517 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.513+0000 7fad25da1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad20108dc0 con 0x7fad200f5410 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad1f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad201088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad1f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad201088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59064/0 (socket says 172.21.15.170:59064) 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad1f7fe700 1 -- 172.21.15.170:0/1783784201 learned_addr learned my addr 172.21.15.170:0/1783784201 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad1f7fe700 1 -- 172.21.15.170:0/1783784201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14005040 con 0x7fad200f5410 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad1f7fe700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad201088d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fad14013510 tx=0x7fad1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad14009320 con 0x7fad200f5410 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad20108fc0 con 0x7fad200f5410 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad14013f00 con 0x7fad200f5410 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad14014d90 con 0x7fad200f5410 2023-12-08T10:13:45.518 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.514+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad20109450 con 0x7fad200f5410 2023-12-08T10:13:45.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.515+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad140263a0 con 0x7fad200f5410 2023-12-08T10:13:45.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.515+0000 7fad07fff700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fad0802ff60 0x7fad08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:45.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.515+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad14041000 con 0x7fad200f5410 2023-12-08T10:13:45.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.515+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad20045ad0 con 0x7fad200f5410 2023-12-08T10:13:45.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.516+0000 7fad1effd700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fad0802ff60 0x7fad08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:45.519 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.516+0000 7fad1effd700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fad0802ff60 0x7fad08032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fad100096a0 tx=0x7fad10006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:45.522 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.520+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad14012920 con 0x7fad200f5410 2023-12-08T10:13:45.728 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.725+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad200faf50 con 0x7fad200f5410 2023-12-08T10:13:45.728 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.726+0000 7fad07fff700 1 -- 172.21.15.170:0/1783784201 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad1402b020 con 0x7fad200f5410 2023-12-08T10:13:45.729 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:45.729 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fad0802ff60 msgr2=0x7fad08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fad0802ff60 0x7fad08032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fad100096a0 tx=0x7fad10006b40).stop 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 msgr2=0x7fad201088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad201088d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fad14013510 tx=0x7fad1400b040).stop 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 shutdown_connections 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fad0802ff60 0x7fad08032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 --2- 172.21.15.170:0/1783784201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad200f5410 0x7fad201088d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.727+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 >> 172.21.15.170:0/1783784201 conn(0x7fad200f0b70 msgr2=0x7fad2005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.728+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 shutdown_connections 2023-12-08T10:13:45.730 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:45.728+0000 7fad25da1700 1 -- 172.21.15.170:0/1783784201 wait complete. 2023-12-08T10:13:45.731 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:46.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:44.375772+0000 mgr.smithi023.tgiigg (mgr.14164) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:46.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:45.727693+0000 mon.smithi023 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.170:0/1783784201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:47.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:47.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 -- 172.21.15.170:0/566950573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 msgr2=0x7f51740f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 --2- 172.21.15.170:0/566950573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f9150 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5164009fb0 tx=0x7f5164009b30).stop 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 -- 172.21.15.170:0/566950573 shutdown_connections 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 --2- 172.21.15.170:0/566950573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f9150 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 -- 172.21.15.170:0/566950573 >> 172.21.15.170:0/566950573 conn(0x7f51740f0b70 msgr2=0x7f51740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 -- 172.21.15.170:0/566950573 shutdown_connections 2023-12-08T10:13:48.209 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 -- 172.21.15.170:0/566950573 wait complete. 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 Processor -- start 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.205+0000 7f517a5b3700 1 -- start start 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f517a5b3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f517a5b3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740f5050 con 0x7f5174060220 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5173fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5173fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59082/0 (socket says 172.21.15.170:59082) 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5173fff700 1 -- 172.21.15.170:0/3344608121 learned_addr learned my addr 172.21.15.170:0/3344608121 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:48.210 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5173fff700 1 -- 172.21.15.170:0/3344608121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5164005040 con 0x7f5174060220 2023-12-08T10:13:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5173fff700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f6bd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5164013510 tx=0x7f516400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5164009320 con 0x7f5174060220 2023-12-08T10:13:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5164013f00 con 0x7f5174060220 2023-12-08T10:13:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51740f51f0 con 0x7f5174060220 2023-12-08T10:13:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.206+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5164014dc0 con 0x7f5174060220 2023-12-08T10:13:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.207+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51740f5680 con 0x7f5174060220 2023-12-08T10:13:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.207+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51640095f0 con 0x7f5174060220 2023-12-08T10:13:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.207+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5174059100 con 0x7f5174060220 2023-12-08T10:13:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.207+0000 7f5170ff9700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f515c02ff60 0x7f515c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.207+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5164041100 con 0x7f5174060220 2023-12-08T10:13:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.208+0000 7f51737fe700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f515c02ff60 0x7f515c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.208+0000 7f51737fe700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f515c02ff60 0x7f515c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f51680096a0 tx=0x7f5168006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:48.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.212+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5164014420 con 0x7f5174060220 2023-12-08T10:13:48.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.419+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51740fbf50 con 0x7f5174060220 2023-12-08T10:13:48.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.420+0000 7f5170ff9700 1 -- 172.21.15.170:0/3344608121 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5164010230 con 0x7f5174060220 2023-12-08T10:13:48.422 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:48.422 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:48.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f515c02ff60 msgr2=0x7f515c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:48.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f515c02ff60 0x7f515c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f51680096a0 tx=0x7f5168006b40).stop 2023-12-08T10:13:48.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 msgr2=0x7f51740f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:48.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f6bd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f5164013510 tx=0x7f516400b040).stop 2023-12-08T10:13:48.424 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 shutdown_connections 2023-12-08T10:13:48.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f515c02ff60 0x7f515c032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:48.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 --2- 172.21.15.170:0/3344608121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5174060220 0x7f51740f6bd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:48.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.421+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 >> 172.21.15.170:0/3344608121 conn(0x7f51740f0b70 msgr2=0x7f51740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:48.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.422+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 shutdown_connections 2023-12-08T10:13:48.425 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:48.422+0000 7f517a5b3700 1 -- 172.21.15.170:0/3344608121 wait complete. 2023-12-08T10:13:48.425 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:48.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:46.376129+0000 mgr.smithi023.tgiigg (mgr.14164) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:49.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:48 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:48.421537+0000 mon.smithi023 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.170:0/3344608121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:49.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:49.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:50.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:48.376425+0000 mgr.smithi023.tgiigg (mgr.14164) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:50.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 -- 172.21.15.170:0/1707827889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 msgr2=0x7f8aec0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:50.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 --2- 172.21.15.170:0/1707827889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec0f90f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f8ae0009fb0 tx=0x7f8ae0009b30).stop 2023-12-08T10:13:50.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 -- 172.21.15.170:0/1707827889 shutdown_connections 2023-12-08T10:13:50.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 --2- 172.21.15.170:0/1707827889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec0f90f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:50.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 -- 172.21.15.170:0/1707827889 >> 172.21.15.170:0/1707827889 conn(0x7f8aec0f0b70 msgr2=0x7f8aec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:50.928 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 -- 172.21.15.170:0/1707827889 shutdown_connections 2023-12-08T10:13:50.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.925+0000 7f8af3ed9700 1 -- 172.21.15.170:0/1707827889 wait complete. 2023-12-08T10:13:50.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.926+0000 7f8af3ed9700 1 Processor -- start 2023-12-08T10:13:50.929 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.926+0000 7f8af3ed9700 1 -- start start 2023-12-08T10:13:50.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.926+0000 7f8af3ed9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:50.930 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.926+0000 7f8af3ed9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec106bd0 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af1c75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af1c75700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59106/0 (socket says 172.21.15.170:59106) 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af1c75700 1 -- 172.21.15.170:0/2298482702 learned_addr learned my addr 172.21.15.170:0/2298482702 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af1c75700 1 -- 172.21.15.170:0/2298482702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0005040 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af1c75700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec1066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8ae0013510 tx=0x7f8ae000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae0009320 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec106dd0 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ae0013f00 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ae0014dc0 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.927+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec107260 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.928+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ae0014420 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.928+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aec045ad0 con 0x7f8aec060280 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.928+0000 7f8ade7fc700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:50.931 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.928+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ae0041050 con 0x7f8aec060280 2023-12-08T10:13:50.932 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.929+0000 7f8af1474700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:50.932 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.929+0000 7f8af1474700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8ad802ff60 0x7f8ad8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8ae80096a0 tx=0x7f8ae8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:50.935 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:50.933+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ae00095f0 con 0x7f8aec060280 2023-12-08T10:13:51.142 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.140+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8aec108110 con 0x7f8aec060280 2023-12-08T10:13:51.143 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.141+0000 7f8ade7fc700 1 -- 172.21.15.170:0/2298482702 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8ae001cb10 con 0x7f8aec060280 2023-12-08T10:13:51.143 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:51.143 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.142+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8ad802ff60 msgr2=0x7f8ad8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.142+0000 7f8af3ed9700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8ad802ff60 0x7f8ad8032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f8ae80096a0 tx=0x7f8ae8006b40).stop 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.142+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 msgr2=0x7f8aec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.142+0000 7f8af3ed9700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec1066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f8ae0013510 tx=0x7f8ae000b040).stop 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.142+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 shutdown_connections 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.143+0000 7f8af3ed9700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f8ad802ff60 0x7f8ad8032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.143+0000 7f8af3ed9700 1 --2- 172.21.15.170:0/2298482702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec060280 0x7f8aec1066e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:51.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.143+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 >> 172.21.15.170:0/2298482702 conn(0x7f8aec0f0b70 msgr2=0x7f8aec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:51.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.143+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 shutdown_connections 2023-12-08T10:13:51.146 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:51.143+0000 7f8af3ed9700 1 -- 172.21.15.170:0/2298482702 wait complete. 2023-12-08T10:13:51.146 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:52.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:50.376703+0000 mgr.smithi023.tgiigg (mgr.14164) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:52.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:51.142613+0000 mon.smithi023 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.170:0/2298482702' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:52.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:52.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.641+0000 7f7d08ac4700 1 -- 172.21.15.170:0/237078883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 msgr2=0x7f7d040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.641+0000 7f7d08ac4700 1 --2- 172.21.15.170:0/237078883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f7cf4009fb0 tx=0x7f7cf4009b30).stop 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 -- 172.21.15.170:0/237078883 shutdown_connections 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 --2- 172.21.15.170:0/237078883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 -- 172.21.15.170:0/237078883 >> 172.21.15.170:0/237078883 conn(0x7f7d040f0b70 msgr2=0x7f7d040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 -- 172.21.15.170:0/237078883 shutdown_connections 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 -- 172.21.15.170:0/237078883 wait complete. 2023-12-08T10:13:53.645 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 Processor -- start 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 -- start start 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d08ac4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040fb820 con 0x7f7d040f5410 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d0259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d0259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44474/0 (socket says 172.21.15.170:44474) 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.642+0000 7f7d0259c700 1 -- 172.21.15.170:0/469870210 learned_addr learned my addr 172.21.15.170:0/469870210 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7d0259c700 1 -- 172.21.15.170:0/469870210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4005040 con 0x7f7d040f5410 2023-12-08T10:13:53.646 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7d0259c700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040fd3a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7cf4013510 tx=0x7f7cf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:53.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf4009320 con 0x7f7d040f5410 2023-12-08T10:13:53.647 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d040fb9c0 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7cf4013f00 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cf4014d90 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.643+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d040fbe50 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.644+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d04059100 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.644+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7cf40095f0 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.644+0000 7f7cfaffd700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7cf002ff60 0x7f7cf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.644+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7cf40410c0 con 0x7f7d040f5410 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.644+0000 7f7d01d9b700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7cf002ff60 0x7f7cf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:53.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.645+0000 7f7d01d9b700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7cf002ff60 0x7f7cf0032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7cec0096a0 tx=0x7f7cec006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:53.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.649+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7cf4014420 con 0x7f7d040f5410 2023-12-08T10:13:53.861 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.857+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d040fae60 con 0x7f7d040f5410 2023-12-08T10:13:53.861 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.858+0000 7f7cfaffd700 1 -- 172.21.15.170:0/469870210 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7cf4010230 con 0x7f7d040f5410 2023-12-08T10:13:53.861 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:53.861 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:53.862 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7cf002ff60 msgr2=0x7f7cf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:53.862 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7cf002ff60 0x7f7cf0032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7cec0096a0 tx=0x7f7cec006b40).stop 2023-12-08T10:13:53.862 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 msgr2=0x7f7d040fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040fd3a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f7cf4013510 tx=0x7f7cf400b040).stop 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 shutdown_connections 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f7cf002ff60 0x7f7cf0032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 --2- 172.21.15.170:0/469870210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d040f5410 0x7f7d040fd3a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 >> 172.21.15.170:0/469870210 conn(0x7f7d040f0b70 msgr2=0x7f7d0405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 shutdown_connections 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:53.860+0000 7f7d08ac4700 1 -- 172.21.15.170:0/469870210 wait complete. 2023-12-08T10:13:53.863 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:52.377067+0000 mgr.smithi023.tgiigg (mgr.14164) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:54.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:53.859783+0000 mon.smithi023 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.170:0/469870210' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:55.245 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:55.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 -- 172.21.15.170:0/3649843618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 msgr2=0x7f9be80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 --2- 172.21.15.170:0/3649843618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be80f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f9bdc009fb0 tx=0x7f9bdc009b30).stop 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 -- 172.21.15.170:0/3649843618 shutdown_connections 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 --2- 172.21.15.170:0/3649843618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be80f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 -- 172.21.15.170:0/3649843618 >> 172.21.15.170:0/3649843618 conn(0x7f9be80f09a0 msgr2=0x7f9be80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 -- 172.21.15.170:0/3649843618 shutdown_connections 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.366+0000 7f9bee978700 1 -- 172.21.15.170:0/3649843618 wait complete. 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bee978700 1 Processor -- start 2023-12-08T10:13:56.369 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bee978700 1 -- start start 2023-12-08T10:13:56.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bee978700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:56.370 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bee978700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81069d0 con 0x7f9be80f51a0 2023-12-08T10:13:56.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bed976700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:56.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bed976700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44492/0 (socket says 172.21.15.170:44492) 2023-12-08T10:13:56.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bed976700 1 -- 172.21.15.170:0/1446458689 learned_addr learned my addr 172.21.15.170:0/1446458689 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:56.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bed976700 1 -- 172.21.15.170:0/1446458689 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bdc005040 con 0x7f9be80f51a0 2023-12-08T10:13:56.371 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.367+0000 7f9bed976700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be81064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f9bdc013510 tx=0x7f9bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bdc009320 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be8106bd0 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bdc013f00 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bdc014dc0 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be8107060 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bdc0095f0 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.368+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be8045ad0 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.369+0000 7f9bda7fc700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bd402ff60 0x7f9bd4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.369+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bdc041000 con 0x7f9be80f51a0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.369+0000 7f9bed175700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bd402ff60 0x7f9bd4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:56.372 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.369+0000 7f9bed175700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bd402ff60 0x7f9bd4032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9be40096a0 tx=0x7f9be4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:56.375 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.373+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bdc014420 con 0x7f9be80f51a0 2023-12-08T10:13:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:54.377407+0000 mgr.smithi023.tgiigg (mgr.14164) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:56.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.578+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9be8107f10 con 0x7f9be80f51a0 2023-12-08T10:13:56.581 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.579+0000 7f9bda7fc700 1 -- 172.21.15.170:0/1446458689 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9bdc010230 con 0x7f9be80f51a0 2023-12-08T10:13:56.582 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:56.582 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.580+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bd402ff60 msgr2=0x7f9bd4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.580+0000 7f9bee978700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bd402ff60 0x7f9bd4032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9be40096a0 tx=0x7f9be4006b40).stop 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.580+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 msgr2=0x7f9be81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.580+0000 7f9bee978700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be81064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f9bdc013510 tx=0x7f9bdc00b040).stop 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.580+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 shutdown_connections 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.580+0000 7f9bee978700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bd402ff60 0x7f9bd4032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.581+0000 7f9bee978700 1 --2- 172.21.15.170:0/1446458689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9be80f51a0 0x7f9be81064e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.581+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 >> 172.21.15.170:0/1446458689 conn(0x7f9be80f09a0 msgr2=0x7f9be805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.581+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 shutdown_connections 2023-12-08T10:13:56.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:56.581+0000 7f9bee978700 1 -- 172.21.15.170:0/1446458689 wait complete. 2023-12-08T10:13:56.584 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:13:57.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:56 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:56.580484+0000 mon.smithi023 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.170:0/1446458689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:13:57.964 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:13:57.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:13:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:13:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:56.377692+0000 mgr.smithi023.tgiigg (mgr.14164) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:13:59.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 -- 172.21.15.170:0/3075092707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 msgr2=0x7f4eb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 --2- 172.21.15.170:0/3075092707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb80f55c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4ea8009fb0 tx=0x7f4ea8009b30).stop 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 -- 172.21.15.170:0/3075092707 shutdown_connections 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 --2- 172.21.15.170:0/3075092707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb80f55c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 -- 172.21.15.170:0/3075092707 >> 172.21.15.170:0/3075092707 conn(0x7f4eb80f09a0 msgr2=0x7f4eb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 -- 172.21.15.170:0/3075092707 shutdown_connections 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 -- 172.21.15.170:0/3075092707 wait complete. 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 Processor -- start 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.099+0000 7f4ebd3be700 1 -- start start 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4ebd3be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:59.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4ebd3be700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb81069d0 con 0x7f4eb80f51a0 2023-12-08T10:13:59.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4eb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:59.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4eb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44510/0 (socket says 172.21.15.170:44510) 2023-12-08T10:13:59.103 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4eb7fff700 1 -- 172.21.15.170:0/2354240566 learned_addr learned my addr 172.21.15.170:0/2354240566 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4eb7fff700 1 -- 172.21.15.170:0/2354240566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea8005040 con 0x7f4eb80f51a0 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.100+0000 7f4eb7fff700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb81064e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4ea8013510 tx=0x7f4ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.101+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea8009320 con 0x7f4eb80f51a0 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.101+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb8106bd0 con 0x7f4eb80f51a0 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.101+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb8107060 con 0x7f4eb80f51a0 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.101+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ea8013f00 con 0x7f4eb80f51a0 2023-12-08T10:13:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.101+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea8014d90 con 0x7f4eb80f51a0 2023-12-08T10:13:59.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.102+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ea80095f0 con 0x7f4eb80f51a0 2023-12-08T10:13:59.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.102+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb8045ad0 con 0x7f4eb80f51a0 2023-12-08T10:13:59.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.102+0000 7f4eb4ff9700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4ea002ff60 0x7f4ea0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:13:59.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.102+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ea8041020 con 0x7f4eb80f51a0 2023-12-08T10:13:59.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.102+0000 7f4eb77fe700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4ea002ff60 0x7f4ea0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:13:59.105 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.102+0000 7f4eb77fe700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4ea002ff60 0x7f4ea0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4eac0096a0 tx=0x7f4eac006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:13:59.109 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.107+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ea8014420 con 0x7f4eb80f51a0 2023-12-08T10:13:59.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.311+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4eb8107f10 con 0x7f4eb80f51a0 2023-12-08T10:13:59.313 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.311+0000 7f4eb4ff9700 1 -- 172.21.15.170:0/2354240566 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ea8012980 con 0x7f4eb80f51a0 2023-12-08T10:13:59.314 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:13:59.314 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4ea002ff60 msgr2=0x7f4ea0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4ea002ff60 0x7f4ea0032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4eac0096a0 tx=0x7f4eac006b40).stop 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 msgr2=0x7f4eb81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb81064e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4ea8013510 tx=0x7f4ea800b040).stop 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 shutdown_connections 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4ea002ff60 0x7f4ea0032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 --2- 172.21.15.170:0/2354240566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4eb80f51a0 0x7f4eb81064e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 >> 172.21.15.170:0/2354240566 conn(0x7f4eb80f09a0 msgr2=0x7f4eb805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 shutdown_connections 2023-12-08T10:13:59.316 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:13:59.313+0000 7f4ebd3be700 1 -- 172.21.15.170:0/2354240566 wait complete. 2023-12-08T10:13:59.317 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:00 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:13:58.378017+0000 mgr.smithi023.tgiigg (mgr.14164) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:00 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:13:59.313145+0000 mon.smithi023 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.170:0/2354240566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:00.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:00.695 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 -- 172.21.15.170:0/1340524108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 msgr2=0x7fdf8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 --2- 172.21.15.170:0/1340524108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c0f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fdf7c009fb0 tx=0x7fdf7c009b30).stop 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 -- 172.21.15.170:0/1340524108 shutdown_connections 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 --2- 172.21.15.170:0/1340524108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c0f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 -- 172.21.15.170:0/1340524108 >> 172.21.15.170:0/1340524108 conn(0x7fdf8c0f1c70 msgr2=0x7fdf8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 -- 172.21.15.170:0/1340524108 shutdown_connections 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.826+0000 7fdf93da5700 1 -- 172.21.15.170:0/1340524108 wait complete. 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf93da5700 1 Processor -- start 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf93da5700 1 -- start start 2023-12-08T10:14:01.832 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf93da5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf93da5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c106c20 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf91b41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf91b41700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44520/0 (socket says 172.21.15.170:44520) 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf91b41700 1 -- 172.21.15.170:0/2103452294 learned_addr learned my addr 172.21.15.170:0/2103452294 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.827+0000 7fdf91b41700 1 -- 172.21.15.170:0/2103452294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c005040 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.828+0000 7fdf91b41700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c106730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fdf7c013510 tx=0x7fdf7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.828+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7c009320 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.828+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8c106e20 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.828+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf7c013f00 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.828+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7c014d90 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.828+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c1072b0 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.829+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf7c0095f0 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.829+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c045ad0 con 0x7fdf8c0f6510 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.829+0000 7fdf827fc700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdf7802ff60 0x7fdf78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:01.833 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.829+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf7c040fd0 con 0x7fdf8c0f6510 2023-12-08T10:14:01.834 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.829+0000 7fdf91340700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdf7802ff60 0x7fdf78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:01.834 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.829+0000 7fdf91340700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdf7802ff60 0x7fdf78032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fdf880096a0 tx=0x7fdf88006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:01.835 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:01.833+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf7c014420 con 0x7fdf8c0f6510 2023-12-08T10:14:02.040 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.038+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf8c0fbf60 con 0x7fdf8c0f6510 2023-12-08T10:14:02.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.039+0000 7fdf827fc700 1 -- 172.21.15.170:0/2103452294 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf7c01d7f0 con 0x7fdf8c0f6510 2023-12-08T10:14:02.041 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:02.041 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdf7802ff60 msgr2=0x7fdf78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdf7802ff60 0x7fdf78032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fdf880096a0 tx=0x7fdf88006b40).stop 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 msgr2=0x7fdf8c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c106730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fdf7c013510 tx=0x7fdf7c00b040).stop 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 shutdown_connections 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdf7802ff60 0x7fdf78032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 --2- 172.21.15.170:0/2103452294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf8c0f6510 0x7fdf8c106730 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 >> 172.21.15.170:0/2103452294 conn(0x7fdf8c0f1c70 msgr2=0x7fdf8c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 shutdown_connections 2023-12-08T10:14:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:02.040+0000 7fdf93da5700 1 -- 172.21.15.170:0/2103452294 wait complete. 2023-12-08T10:14:02.044 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:00.378390+0000 mgr.smithi023.tgiigg (mgr.14164) 235 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:03.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:03.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:03.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:02.040369+0000 mon.smithi023 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.170:0/2103452294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:04 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:02.378600+0000 mgr.smithi023.tgiigg (mgr.14164) 236 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/2510380695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 msgr2=0x7f79680f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 --2- 172.21.15.170:0/2510380695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79680f90f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f7958009fb0 tx=0x7f7958009b30).stop 2023-12-08T10:14:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/2510380695 shutdown_connections 2023-12-08T10:14:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 --2- 172.21.15.170:0/2510380695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79680f90f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:04.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/2510380695 >> 172.21.15.170:0/2510380695 conn(0x7f79680f0b70 msgr2=0x7f79680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/2510380695 shutdown_connections 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/2510380695 wait complete. 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 Processor -- start 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- start start 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79681066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7968106bd0 con 0x7f7968060280 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f7966ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79681066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f7966ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79681066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:38792/0 (socket says 172.21.15.170:38792) 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f7966ffd700 1 -- 172.21.15.170:0/3984108627 learned_addr learned my addr 172.21.15.170:0/3984108627 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f7966ffd700 1 -- 172.21.15.170:0/3984108627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7958005040 con 0x7f7968060280 2023-12-08T10:14:04.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f7966ffd700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79681066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f7958013510 tx=0x7f795800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7958009320 con 0x7f7968060280 2023-12-08T10:14:04.637 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7968106dd0 con 0x7f7968060280 2023-12-08T10:14:04.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7958013f00 con 0x7f7968060280 2023-12-08T10:14:04.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7958014dc0 con 0x7f7968060280 2023-12-08T10:14:04.638 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.633+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7968107260 con 0x7f7968060280 2023-12-08T10:14:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.634+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79580095f0 con 0x7f7968060280 2023-12-08T10:14:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.634+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968045ad0 con 0x7f7968060280 2023-12-08T10:14:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.634+0000 7f794f7fe700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f795002ff60 0x7f7950032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.634+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79580410e0 con 0x7f7968060280 2023-12-08T10:14:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.635+0000 7f79667fc700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f795002ff60 0x7f7950032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.635+0000 7f79667fc700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f795002ff60 0x7f7950032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f795c0096a0 tx=0x7f795c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:04.641 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.640+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7958014420 con 0x7f7968060280 2023-12-08T10:14:04.845 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.843+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79681080b0 con 0x7f7968060280 2023-12-08T10:14:04.846 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f794f7fe700 1 -- 172.21.15.170:0/3984108627 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f795801c920 con 0x7f7968060280 2023-12-08T10:14:04.846 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:04.846 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:04.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f795002ff60 msgr2=0x7f7950032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:04.847 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f795002ff60 0x7f7950032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f795c0096a0 tx=0x7f795c006b40).stop 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 msgr2=0x7f79681066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79681066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f7958013510 tx=0x7f795800b040).stop 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 shutdown_connections 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f795002ff60 0x7f7950032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 --2- 172.21.15.170:0/3984108627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968060280 0x7f79681066e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 >> 172.21.15.170:0/3984108627 conn(0x7f79680f0b70 msgr2=0x7f79680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 shutdown_connections 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:04.845+0000 7f796d7a4700 1 -- 172.21.15.170:0/3984108627 wait complete. 2023-12-08T10:14:04.848 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:05.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:04.845301+0000 mon.smithi023 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.170:0/3984108627' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:06.263 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:06.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:06 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:04.378955+0000 mgr.smithi023.tgiigg (mgr.14164) 237 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.417+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3407831176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 msgr2=0x7f80e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.417+0000 7f80e5bf3700 1 --2- 172.21.15.170:0/3407831176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e00f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f80d4009fb0 tx=0x7f80d4009b30).stop 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.417+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3407831176 shutdown_connections 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.417+0000 7f80e5bf3700 1 --2- 172.21.15.170:0/3407831176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e00f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.417+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3407831176 >> 172.21.15.170:0/3407831176 conn(0x7f80e00f1c70 msgr2=0x7f80e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.417+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3407831176 shutdown_connections 2023-12-08T10:14:07.419 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.418+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3407831176 wait complete. 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.418+0000 7f80e5bf3700 1 Processor -- start 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.418+0000 7f80e5bf3700 1 -- start start 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.418+0000 7f80e5bf3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.418+0000 7f80e5bf3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e0106c20 con 0x7f80e00f6510 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80df7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:38810/0 (socket says 172.21.15.170:38810) 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80df7fe700 1 -- 172.21.15.170:0/3041052548 learned_addr learned my addr 172.21.15.170:0/3041052548 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80df7fe700 1 -- 172.21.15.170:0/3041052548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d4005040 con 0x7f80e00f6510 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80df7fe700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e0106730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f80d4013510 tx=0x7f80d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80d4009320 con 0x7f80e00f6510 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e0106e20 con 0x7f80e00f6510 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80d4013f00 con 0x7f80e00f6510 2023-12-08T10:14:07.421 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80d4014d90 con 0x7f80e00f6510 2023-12-08T10:14:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.419+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e01072b0 con 0x7f80e00f6510 2023-12-08T10:14:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.420+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80d40263a0 con 0x7f80e00f6510 2023-12-08T10:14:07.422 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.420+0000 7f80c7fff700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80c802ff60 0x7f80c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:07.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.420+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80d4041040 con 0x7f80e00f6510 2023-12-08T10:14:07.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.420+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e0045ad0 con 0x7f80e00f6510 2023-12-08T10:14:07.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.420+0000 7f80deffd700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80c802ff60 0x7f80c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:07.423 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.421+0000 7f80deffd700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80c802ff60 0x7f80c8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f80d00096a0 tx=0x7f80d0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:07.426 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.425+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80d4012920 con 0x7f80e00f6510 2023-12-08T10:14:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.631+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80e00fbf60 con 0x7f80e00f6510 2023-12-08T10:14:07.633 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.632+0000 7f80c7fff700 1 -- 172.21.15.170:0/3041052548 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80d402b020 con 0x7f80e00f6510 2023-12-08T10:14:07.634 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:07.634 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80c802ff60 msgr2=0x7f80c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80c802ff60 0x7f80c8032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f80d00096a0 tx=0x7f80d0006b40).stop 2023-12-08T10:14:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 msgr2=0x7f80e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e0106730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f80d4013510 tx=0x7f80d400b040).stop 2023-12-08T10:14:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 shutdown_connections 2023-12-08T10:14:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f80c802ff60 0x7f80c8032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 --2- 172.21.15.170:0/3041052548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80e00f6510 0x7f80e0106730 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 >> 172.21.15.170:0/3041052548 conn(0x7f80e00f1c70 msgr2=0x7f80e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 shutdown_connections 2023-12-08T10:14:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:07.634+0000 7f80e5bf3700 1 -- 172.21.15.170:0/3041052548 wait complete. 2023-12-08T10:14:07.636 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:08 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:06.379359+0000 mgr.smithi023.tgiigg (mgr.14164) 238 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:08 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:07.632855+0000 mon.smithi023 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.170:0/3041052548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:09.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:09.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:10.169 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 -- 172.21.15.170:0/410446439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 msgr2=0x7f937c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 --2- 172.21.15.170:0/410446439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c0f55c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f9370009fb0 tx=0x7f9370009b30).stop 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 -- 172.21.15.170:0/410446439 shutdown_connections 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 --2- 172.21.15.170:0/410446439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c0f55c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 -- 172.21.15.170:0/410446439 >> 172.21.15.170:0/410446439 conn(0x7f937c0f09a0 msgr2=0x7f937c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 -- 172.21.15.170:0/410446439 shutdown_connections 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.167+0000 7f9380eea700 1 -- 172.21.15.170:0/410446439 wait complete. 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f9380eea700 1 Processor -- start 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f9380eea700 1 -- start start 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f9380eea700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f9380eea700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c1058a0 con 0x7f937c0f51a0 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f937b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:10.170 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f937b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:38832/0 (socket says 172.21.15.170:38832) 2023-12-08T10:14:10.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.168+0000 7f937b7fe700 1 -- 172.21.15.170:0/2734151287 learned_addr learned my addr 172.21.15.170:0/2734151287 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:10.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f937b7fe700 1 -- 172.21.15.170:0/2734151287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9370005040 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f937b7fe700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c1053b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f9370013510 tx=0x7f937000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9370009320 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c105aa0 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9370013f00 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c105f30 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.169+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9370014dc0 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.170+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c045ad0 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.170+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9370014420 con 0x7f937c0f51a0 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.170+0000 7f9363fff700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f936402ff60 0x7f9364032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:10.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.170+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9370041050 con 0x7f937c0f51a0 2023-12-08T10:14:10.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.170+0000 7f937affd700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f936402ff60 0x7f9364032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:10.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.171+0000 7f937affd700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f936402ff60 0x7f9364032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f936c0096a0 tx=0x7f936c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:10.176 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.175+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93700095f0 con 0x7f937c0f51a0 2023-12-08T10:14:10.380 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.379+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f937c106d80 con 0x7f937c0f51a0 2023-12-08T10:14:10.381 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.380+0000 7f9363fff700 1 -- 172.21.15.170:0/2734151287 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f937001cb10 con 0x7f937c0f51a0 2023-12-08T10:14:10.381 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:10.381 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:10.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f936402ff60 msgr2=0x7f9364032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:10.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f936402ff60 0x7f9364032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f936c0096a0 tx=0x7f936c006b40).stop 2023-12-08T10:14:10.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 msgr2=0x7f937c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:10.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c1053b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f9370013510 tx=0x7f937000b040).stop 2023-12-08T10:14:10.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 shutdown_connections 2023-12-08T10:14:10.383 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f936402ff60 0x7f9364032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:10.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 --2- 172.21.15.170:0/2734151287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f937c0f51a0 0x7f937c1053b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:10.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.381+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 >> 172.21.15.170:0/2734151287 conn(0x7f937c0f09a0 msgr2=0x7f937c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:10.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.382+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 shutdown_connections 2023-12-08T10:14:10.384 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:10.382+0000 7f9380eea700 1 -- 172.21.15.170:0/2734151287 wait complete. 2023-12-08T10:14:10.384 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:08.379752+0000 mgr.smithi023.tgiigg (mgr.14164) 239 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:11.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:10.380501+0000 mon.smithi023 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.170:0/2734151287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:11.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:11.751 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:12.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:12 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:10.380084+0000 mgr.smithi023.tgiigg (mgr.14164) 240 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:12.885 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 -- 172.21.15.170:0/998250506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc0f5410 msgr2=0x7f31bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 --2- 172.21.15.170:0/998250506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc0f5410 0x7f31bc0f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f31b0009fb0 tx=0x7f31b0009b30).stop 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 -- 172.21.15.170:0/998250506 shutdown_connections 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 --2- 172.21.15.170:0/998250506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc0f5410 0x7f31bc0f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 -- 172.21.15.170:0/998250506 >> 172.21.15.170:0/998250506 conn(0x7f31bc0f0b70 msgr2=0x7f31bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 -- 172.21.15.170:0/998250506 shutdown_connections 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 -- 172.21.15.170:0/998250506 wait complete. 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 Processor -- start 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.883+0000 7f31c3014700 1 -- start start 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c3014700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 0x7f31bc105b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c3014700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc106050 con 0x7f31bc105740 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c0db0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 0x7f31bc105b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c0db0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 0x7f31bc105b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48418/0 (socket says 172.21.15.170:48418) 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c0db0700 1 -- 172.21.15.170:0/1468563184 learned_addr learned my addr 172.21.15.170:0/1468563184 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:12.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c0db0700 1 -- 172.21.15.170:0/1468563184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0005040 con 0x7f31bc105740 2023-12-08T10:14:12.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.884+0000 7f31c0db0700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 0x7f31bc105b60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f31b0009f80 tx=0x7f31b0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31b00136c0 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc106250 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31b00095f0 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc106da0 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31b0014420 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc059100 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.885+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31b0025060 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.886+0000 7f31b97fa700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f31ac02ff60 0x7f31ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.886+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31b0040e60 con 0x7f31bc105740 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.886+0000 7f31bbfff700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f31ac02ff60 0x7f31ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:12.888 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.886+0000 7f31bbfff700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f31ac02ff60 0x7f31ac032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f31a80096a0 tx=0x7f31a8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:12.891 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:12.890+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31b001c950 con 0x7f31bc105740 2023-12-08T10:14:13.099 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.097+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31bc107560 con 0x7f31bc105740 2023-12-08T10:14:13.100 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.098+0000 7f31b97fa700 1 -- 172.21.15.170:0/1468563184 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31b001cb60 con 0x7f31bc105740 2023-12-08T10:14:13.100 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:13.101 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:13.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f31ac02ff60 msgr2=0x7f31ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:13.101 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f31ac02ff60 0x7f31ac032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f31a80096a0 tx=0x7f31a8006b40).stop 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 msgr2=0x7f31bc105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 0x7f31bc105b60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f31b0009f80 tx=0x7f31b0009b30).stop 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 shutdown_connections 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f31ac02ff60 0x7f31ac032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 --2- 172.21.15.170:0/1468563184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31bc105740 0x7f31bc105b60 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 >> 172.21.15.170:0/1468563184 conn(0x7f31bc0f0b70 msgr2=0x7f31bc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 shutdown_connections 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:13.100+0000 7f31c3014700 1 -- 172.21.15.170:0/1468563184 wait complete. 2023-12-08T10:14:13.102 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:14.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:12.380450+0000 mgr.smithi023.tgiigg (mgr.14164) 241 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:14.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:13.098876+0000 mon.smithi023 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.170:0/1468563184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:14.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:14.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:15.679 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 -- 172.21.15.170:0/1009928321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 msgr2=0x7f43880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:15.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 --2- 172.21.15.170:0/1009928321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f43880f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f437c009fb0 tx=0x7f437c009b30).stop 2023-12-08T10:14:15.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 -- 172.21.15.170:0/1009928321 shutdown_connections 2023-12-08T10:14:15.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 --2- 172.21.15.170:0/1009928321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f43880f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:15.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 -- 172.21.15.170:0/1009928321 >> 172.21.15.170:0/1009928321 conn(0x7f43880f1c70 msgr2=0x7f43880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:15.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 -- 172.21.15.170:0/1009928321 shutdown_connections 2023-12-08T10:14:15.680 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.674+0000 7f438d02f700 1 -- 172.21.15.170:0/1009928321 wait complete. 2023-12-08T10:14:15.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.675+0000 7f438d02f700 1 Processor -- start 2023-12-08T10:14:15.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.675+0000 7f438d02f700 1 -- start start 2023-12-08T10:14:15.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.675+0000 7f438d02f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f4388106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:15.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.675+0000 7f438d02f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388106c20 con 0x7f43880f6510 2023-12-08T10:14:15.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f4386d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f4388106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:15.681 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f4386d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f4388106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48440/0 (socket says 172.21.15.170:48440) 2023-12-08T10:14:15.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f4386d9d700 1 -- 172.21.15.170:0/4019838778 learned_addr learned my addr 172.21.15.170:0/4019838778 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:15.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f4386d9d700 1 -- 172.21.15.170:0/4019838778 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c005040 con 0x7f43880f6510 2023-12-08T10:14:15.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f4386d9d700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f4388106730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f437c013510 tx=0x7f437c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:15.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f437c009320 con 0x7f43880f6510 2023-12-08T10:14:15.682 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4388106e20 con 0x7f43880f6510 2023-12-08T10:14:15.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43881072b0 con 0x7f43880f6510 2023-12-08T10:14:15.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f437c013f00 con 0x7f43880f6510 2023-12-08T10:14:15.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.676+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f437c014d60 con 0x7f43880f6510 2023-12-08T10:14:15.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.677+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f437c0263a0 con 0x7f43880f6510 2023-12-08T10:14:15.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.677+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4388045ad0 con 0x7f43880f6510 2023-12-08T10:14:15.683 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.677+0000 7f436f7fe700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f437002ff60 0x7f4370032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:15.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.677+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f437c0412d0 con 0x7f43880f6510 2023-12-08T10:14:15.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.677+0000 7f438659c700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f437002ff60 0x7f4370032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:15.684 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.678+0000 7f438659c700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f437002ff60 0x7f4370032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f43780096a0 tx=0x7f4378006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:15.685 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.682+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f437c012330 con 0x7f43880f6510 2023-12-08T10:14:15.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.885+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43880fbf60 con 0x7f43880f6510 2023-12-08T10:14:15.887 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.886+0000 7f436f7fe700 1 -- 172.21.15.170:0/4019838778 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f437c01cd80 con 0x7f43880f6510 2023-12-08T10:14:15.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:15.887 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f437002ff60 msgr2=0x7f4370032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f437002ff60 0x7f4370032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f43780096a0 tx=0x7f4378006b40).stop 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 msgr2=0x7f4388106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f4388106730 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f437c013510 tx=0x7f437c00b040).stop 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 shutdown_connections 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f437002ff60 0x7f4370032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 --2- 172.21.15.170:0/4019838778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43880f6510 0x7f4388106730 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 >> 172.21.15.170:0/4019838778 conn(0x7f43880f1c70 msgr2=0x7f43880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.887+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 shutdown_connections 2023-12-08T10:14:15.889 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:15.888+0000 7f438d02f700 1 -- 172.21.15.170:0/4019838778 wait complete. 2023-12-08T10:14:15.890 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:16.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:14.380755+0000 mgr.smithi023.tgiigg (mgr.14164) 242 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:16.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:15.886580+0000 mon.smithi023 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.170:0/4019838778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:17.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:17.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 -- 172.21.15.170:0/59668624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 msgr2=0x7f44f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 --2- 172.21.15.170:0/59668624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f40f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f44e4009fb0 tx=0x7f44e4009b30).stop 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 -- 172.21.15.170:0/59668624 shutdown_connections 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 --2- 172.21.15.170:0/59668624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f40f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 -- 172.21.15.170:0/59668624 >> 172.21.15.170:0/59668624 conn(0x7f44f40f1c70 msgr2=0x7f44f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 -- 172.21.15.170:0/59668624 shutdown_connections 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 -- 172.21.15.170:0/59668624 wait complete. 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.384+0000 7f44fbc26700 1 Processor -- start 2023-12-08T10:14:18.387 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44fbc26700 1 -- start start 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44fbc26700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44fbc26700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f4106c20 con 0x7f44f40f6510 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44f99c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44f99c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48454/0 (socket says 172.21.15.170:48454) 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44f99c2700 1 -- 172.21.15.170:0/706140645 learned_addr learned my addr 172.21.15.170:0/706140645 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44f99c2700 1 -- 172.21.15.170:0/706140645 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4005040 con 0x7f44f40f6510 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44f99c2700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f4106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f44e4013510 tx=0x7f44e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.385+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e4009320 con 0x7f44f40f6510 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.386+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f4106e20 con 0x7f44f40f6510 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.386+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44e4013f00 con 0x7f44f40f6510 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.386+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e4014d90 con 0x7f44f40f6510 2023-12-08T10:14:18.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.386+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f41072b0 con 0x7f44f40f6510 2023-12-08T10:14:18.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.386+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f4045ad0 con 0x7f44f40f6510 2023-12-08T10:14:18.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.387+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44e40095f0 con 0x7f44f40f6510 2023-12-08T10:14:18.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.387+0000 7f44ea7fc700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f44e002ff60 0x7f44e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:18.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.387+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44e4040fc0 con 0x7f44f40f6510 2023-12-08T10:14:18.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.387+0000 7f44f91c1700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f44e002ff60 0x7f44e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:18.389 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.387+0000 7f44f91c1700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f44e002ff60 0x7f44e0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f44f00096a0 tx=0x7f44f0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:18.394 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.391+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44e4014420 con 0x7f44f40f6510 2023-12-08T10:14:18.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:16.381078+0000 mgr.smithi023.tgiigg (mgr.14164) 243 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:18.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.597+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44f40fbf60 con 0x7f44f40f6510 2023-12-08T10:14:18.599 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.598+0000 7f44ea7fc700 1 -- 172.21.15.170:0/706140645 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44e4010230 con 0x7f44f40f6510 2023-12-08T10:14:18.600 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:18.600 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f44e002ff60 msgr2=0x7f44e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f44e002ff60 0x7f44e0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f44f00096a0 tx=0x7f44f0006b40).stop 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 msgr2=0x7f44f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f4106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f44e4013510 tx=0x7f44e400b040).stop 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 shutdown_connections 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f44e002ff60 0x7f44e0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 --2- 172.21.15.170:0/706140645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f40f6510 0x7f44f4106730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 >> 172.21.15.170:0/706140645 conn(0x7f44f40f1c70 msgr2=0x7f44f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 shutdown_connections 2023-12-08T10:14:18.602 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:18.600+0000 7f44fbc26700 1 -- 172.21.15.170:0/706140645 wait complete. 2023-12-08T10:14:18.603 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:19.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:18.598987+0000 mon.smithi023 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.170:0/706140645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:19.968 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:19.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:20.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:18.381412+0000 mgr.smithi023.tgiigg (mgr.14164) 244 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:20.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:14:18.723542+0000 mgr.smithi023.tgiigg (mgr.14164) 245 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi023 ... 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Verifying port 9100 ... 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Non-zero exit code 1 from systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: systemctl: stderr Job for ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023.service failed because a timeout was exceeded. 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: systemctl: stderr See "systemctl status ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023.service" and "journalctl -xe" for details. 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Traceback (most recent call last): 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: main() 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: r = ctx.func(ctx) 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: return func(ctx) 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-08T10:14:20.433 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ports=daemon_ports) 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: RuntimeError: Failed command: systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Traceback (most recent call last): 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: yield (conn, connr) 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: code, '\n'.join(err))) 2023-12-08T10:14:20.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi023 ... 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Verifying port 9100 ... 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Non-zero exit code 1 from systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: systemctl: stderr Job for ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023.service failed because a timeout was exceeded. 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: systemctl: stderr See "systemctl status ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023.service" and "journalctl -xe" for details. 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: Traceback (most recent call last): 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: main() 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: r = ctx.func(ctx) 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: return func(ctx) 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: ports=daemon_ports) 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-08T10:14:20.435 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-08T10:14:20.437 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-08T10:14:20.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-08T10:14:20.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: File "/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-08T10:14:20.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-08T10:14:20.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: RuntimeError: Failed command: systemctl start ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@node-exporter.smithi023 2023-12-08T10:14:20.438 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:20 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cephadm 2023-12-08T10:14:18.729317+0000 mgr.smithi023.tgiigg (mgr.14164) 246 : cephadm [INF] Deploying daemon prometheus.smithi023 on smithi023 2023-12-08T10:14:21.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.078+0000 7fe9a5a08700 1 -- 172.21.15.170:0/956186642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 msgr2=0x7fe9a00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:21.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.078+0000 7fe9a5a08700 1 --2- 172.21.15.170:0/956186642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a00fb570 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fe988009fb0 tx=0x7fe988009b30).stop 2023-12-08T10:14:21.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.078+0000 7fe9a5a08700 1 -- 172.21.15.170:0/956186642 shutdown_connections 2023-12-08T10:14:21.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.078+0000 7fe9a5a08700 1 --2- 172.21.15.170:0/956186642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a00fb570 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:21.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.078+0000 7fe9a5a08700 1 -- 172.21.15.170:0/956186642 >> 172.21.15.170:0/956186642 conn(0x7fe9a00f0b70 msgr2=0x7fe9a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:21.082 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.079+0000 7fe9a5a08700 1 -- 172.21.15.170:0/956186642 shutdown_connections 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.079+0000 7fe9a5a08700 1 -- 172.21.15.170:0/956186642 wait complete. 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.079+0000 7fe9a5a08700 1 Processor -- start 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.079+0000 7fe9a5a08700 1 -- start start 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.079+0000 7fe9a5a08700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.079+0000 7fe9a5a08700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a0108dc0 con 0x7fe9a0060200 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe99effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe99effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:48470/0 (socket says 172.21.15.170:48470) 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe99effd700 1 -- 172.21.15.170:0/324149062 learned_addr learned my addr 172.21.15.170:0/324149062 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:21.083 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe99effd700 1 -- 172.21.15.170:0/324149062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988005040 con 0x7fe9a0060200 2023-12-08T10:14:21.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe99effd700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a01088d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fe988013510 tx=0x7fe98800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:21.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe988009320 con 0x7fe9a0060200 2023-12-08T10:14:21.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a0108fc0 con 0x7fe9a0060200 2023-12-08T10:14:21.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe988013f00 con 0x7fe9a0060200 2023-12-08T10:14:21.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe988014de0 con 0x7fe9a0060200 2023-12-08T10:14:21.084 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.080+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a0109450 con 0x7fe9a0060200 2023-12-08T10:14:21.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.081+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9880095f0 con 0x7fe9a0060200 2023-12-08T10:14:21.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.081+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a0045ad0 con 0x7fe9a0060200 2023-12-08T10:14:21.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.081+0000 7fe9977fe700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe98c02ff60 0x7fe98c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:21.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.081+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe988040fe0 con 0x7fe9a0060200 2023-12-08T10:14:21.085 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.081+0000 7fe99e7fc700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe98c02ff60 0x7fe98c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:21.086 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.082+0000 7fe99e7fc700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe98c02ff60 0x7fe98c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe9900096a0 tx=0x7fe990006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:21.087 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.086+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe98801c920 con 0x7fe9a0060200 2023-12-08T10:14:21.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.289+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9a010a300 con 0x7fe9a0060200 2023-12-08T10:14:21.291 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.290+0000 7fe9977fe700 1 -- 172.21.15.170:0/324149062 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe98801c920 con 0x7fe9a0060200 2023-12-08T10:14:21.292 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:21.292 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe98c02ff60 msgr2=0x7fe98c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe98c02ff60 0x7fe98c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe9900096a0 tx=0x7fe990006b40).stop 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 msgr2=0x7fe9a01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a01088d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fe988013510 tx=0x7fe98800b040).stop 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 shutdown_connections 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fe98c02ff60 0x7fe98c032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 --2- 172.21.15.170:0/324149062 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0060200 0x7fe9a01088d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 >> 172.21.15.170:0/324149062 conn(0x7fe9a00f0b70 msgr2=0x7fe9a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:21.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 shutdown_connections 2023-12-08T10:14:21.294 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:21.292+0000 7fe9a5a08700 1 -- 172.21.15.170:0/324149062 wait complete. 2023-12-08T10:14:21.294 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:22.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:20.381760+0000 mgr.smithi023.tgiigg (mgr.14164) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:22.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:22 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:21.290786+0000 mon.smithi023 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.170:0/324149062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:22.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:22.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:22.456903+0000 mon.smithi023 (mon.0) 304 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/mirror_snapshot_schedule"}]: dispatch 2023-12-08T10:14:23.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:23 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:22.458256+0000 mon.smithi023 (mon.0) 305 : audit [INF] from='mgr.14164 172.21.15.23:0/765389117' entity='mgr.smithi023.tgiigg' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi023.tgiigg/trash_purge_schedule"}]: dispatch 2023-12-08T10:14:23.848 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 -- 172.21.15.170:0/556287115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 msgr2=0x7fd1c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 --2- 172.21.15.170:0/556287115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c00f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd1b4009fb0 tx=0x7fd1b4009b30).stop 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 -- 172.21.15.170:0/556287115 shutdown_connections 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 --2- 172.21.15.170:0/556287115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c00f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 -- 172.21.15.170:0/556287115 >> 172.21.15.170:0/556287115 conn(0x7fd1c00f0b70 msgr2=0x7fd1c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 -- 172.21.15.170:0/556287115 shutdown_connections 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.846+0000 7fd1c87e1700 1 -- 172.21.15.170:0/556287115 wait complete. 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c87e1700 1 Processor -- start 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c87e1700 1 -- start start 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c87e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c87e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0105b20 con 0x7fd1c00f5410 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c657d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c657d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44404/0 (socket says 172.21.15.170:44404) 2023-12-08T10:14:23.849 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.847+0000 7fd1c657d700 1 -- 172.21.15.170:0/4165369478 learned_addr learned my addr 172.21.15.170:0/4165369478 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:23.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1c657d700 1 -- 172.21.15.170:0/4165369478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4005040 con 0x7fd1c00f5410 2023-12-08T10:14:23.850 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1c657d700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c0105630 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd1b4013510 tx=0x7fd1b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1b4009320 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c0105d20 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1b4013f00 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1b4014d90 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.848+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c01061b0 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.849+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1b40095f0 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.849+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c0045ad0 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.849+0000 7fd1b2ffd700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd1ac02ff60 0x7fd1ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.849+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1b4040fc0 con 0x7fd1c00f5410 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.849+0000 7fd1c5d7c700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd1ac02ff60 0x7fd1ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:23.851 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.849+0000 7fd1c5d7c700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd1ac02ff60 0x7fd1ac032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fd1bc0096a0 tx=0x7fd1bc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:23.854 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:23.853+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1b4014420 con 0x7fd1c00f5410 2023-12-08T10:14:24.060 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.059+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1c00fae60 con 0x7fd1c00f5410 2023-12-08T10:14:24.061 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.060+0000 7fd1b2ffd700 1 -- 172.21.15.170:0/4165369478 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1b4010230 con 0x7fd1c00f5410 2023-12-08T10:14:24.061 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:24.061 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:24.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd1ac02ff60 msgr2=0x7fd1ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:24.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd1ac02ff60 0x7fd1ac032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fd1bc0096a0 tx=0x7fd1bc006b40).stop 2023-12-08T10:14:24.063 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 msgr2=0x7fd1c0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c0105630 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd1b4013510 tx=0x7fd1b400b040).stop 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 shutdown_connections 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd1ac02ff60 0x7fd1ac032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 --2- 172.21.15.170:0/4165369478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1c00f5410 0x7fd1c0105630 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 >> 172.21.15.170:0/4165369478 conn(0x7fd1c00f0b70 msgr2=0x7fd1c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 shutdown_connections 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:24.061+0000 7fd1c87e1700 1 -- 172.21.15.170:0/4165369478 wait complete. 2023-12-08T10:14:24.064 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:24.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:22.382115+0000 mgr.smithi023.tgiigg (mgr.14164) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:24.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:24 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:24.060561+0000 mon.smithi023 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.170:0/4165369478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:25.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:25.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:26.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:26 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:24.382418+0000 mgr.smithi023.tgiigg (mgr.14164) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 -- 172.21.15.170:0/4115902861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640f51a0 msgr2=0x7ff6640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 --2- 172.21.15.170:0/4115902861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640f51a0 0x7ff6640f55c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ff658009fb0 tx=0x7ff658009b30).stop 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 -- 172.21.15.170:0/4115902861 shutdown_connections 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 --2- 172.21.15.170:0/4115902861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640f51a0 0x7ff6640f55c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 -- 172.21.15.170:0/4115902861 >> 172.21.15.170:0/4115902861 conn(0x7ff6640f09a0 msgr2=0x7ff6640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 -- 172.21.15.170:0/4115902861 shutdown_connections 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 -- 172.21.15.170:0/4115902861 wait complete. 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.577+0000 7ff669d71700 1 Processor -- start 2023-12-08T10:14:26.580 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff669d71700 1 -- start start 2023-12-08T10:14:26.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff669d71700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 0x7ff664105920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:26.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff669d71700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664105e10 con 0x7ff664105500 2023-12-08T10:14:26.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff668d6f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 0x7ff664105920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:26.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff668d6f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 0x7ff664105920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44422/0 (socket says 172.21.15.170:44422) 2023-12-08T10:14:26.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff668d6f700 1 -- 172.21.15.170:0/402734215 learned_addr learned my addr 172.21.15.170:0/402734215 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:26.582 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.578+0000 7ff668d6f700 1 -- 172.21.15.170:0/402734215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff658005040 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.579+0000 7ff668d6f700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 0x7ff664105920 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ff658009f80 tx=0x7ff658009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.579+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6580136c0 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.579+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff664106010 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.579+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6580095f0 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.579+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff658014420 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.579+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff664106b60 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.580+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff658025060 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.580+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664059100 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.580+0000 7ff6617fa700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff65402ff60 0x7ff654032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.580+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff658040d80 con 0x7ff664105500 2023-12-08T10:14:26.583 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.580+0000 7ff663fff700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff65402ff60 0x7ff654032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:26.584 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.581+0000 7ff663fff700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff65402ff60 0x7ff654032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7ff6500096a0 tx=0x7ff650006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:26.589 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.585+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff65801c950 con 0x7ff664105500 2023-12-08T10:14:26.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.790+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff664107320 con 0x7ff664105500 2023-12-08T10:14:26.793 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.791+0000 7ff6617fa700 1 -- 172.21.15.170:0/402734215 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff658042090 con 0x7ff664105500 2023-12-08T10:14:26.794 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:26.794 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff65402ff60 msgr2=0x7ff654032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff65402ff60 0x7ff654032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7ff6500096a0 tx=0x7ff650006b40).stop 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 msgr2=0x7ff664105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 0x7ff664105920 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ff658009f80 tx=0x7ff658009b30).stop 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 shutdown_connections 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ff65402ff60 0x7ff654032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 --2- 172.21.15.170:0/402734215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff664105500 0x7ff664105920 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 >> 172.21.15.170:0/402734215 conn(0x7ff6640f09a0 msgr2=0x7ff6640f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 shutdown_connections 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:26.793+0000 7ff669d71700 1 -- 172.21.15.170:0/402734215 wait complete. 2023-12-08T10:14:26.799 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:27.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:27 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:26.793088+0000 mon.smithi023 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.170:0/402734215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:28.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:28.183 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:28.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:28 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:26.382753+0000 mgr.smithi023.tgiigg (mgr.14164) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:29.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 -- 172.21.15.170:0/3297209614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 msgr2=0x7f16e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:29.276 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 --2- 172.21.15.170:0/3297209614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e80f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f16d8009fb0 tx=0x7f16d8009b30).stop 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 -- 172.21.15.170:0/3297209614 shutdown_connections 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 --2- 172.21.15.170:0/3297209614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e80f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 -- 172.21.15.170:0/3297209614 >> 172.21.15.170:0/3297209614 conn(0x7f16e80f1c70 msgr2=0x7f16e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 -- 172.21.15.170:0/3297209614 shutdown_connections 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.272+0000 7f16f0426700 1 -- 172.21.15.170:0/3297209614 wait complete. 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16f0426700 1 Processor -- start 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16f0426700 1 -- start start 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16f0426700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16f0426700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e8106c20 con 0x7f16e80f6510 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16ee1c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16ee1c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44450/0 (socket says 172.21.15.170:44450) 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.273+0000 7f16ee1c2700 1 -- 172.21.15.170:0/2999930098 learned_addr learned my addr 172.21.15.170:0/2999930098 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16ee1c2700 1 -- 172.21.15.170:0/2999930098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8005040 con 0x7f16e80f6510 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16ee1c2700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f16d8013510 tx=0x7f16d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:29.277 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d8009320 con 0x7f16e80f6510 2023-12-08T10:14:29.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e8106e20 con 0x7f16e80f6510 2023-12-08T10:14:29.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16d8013f00 con 0x7f16e80f6510 2023-12-08T10:14:29.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d8014d90 con 0x7f16e80f6510 2023-12-08T10:14:29.278 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.274+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e81072b0 con 0x7f16e80f6510 2023-12-08T10:14:29.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.275+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16d8014420 con 0x7f16e80f6510 2023-12-08T10:14:29.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.275+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e8045ad0 con 0x7f16e80f6510 2023-12-08T10:14:29.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.275+0000 7f16deffd700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:29.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.275+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16d8041110 con 0x7f16e80f6510 2023-12-08T10:14:29.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.275+0000 7f16ed9c1700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:29.279 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.276+0000 7f16ed9c1700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f16d402ff60 0x7f16d4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f16e40096a0 tx=0x7f16e4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:29.282 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.280+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16d80095f0 con 0x7f16e80f6510 2023-12-08T10:14:29.486 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.484+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16e80fbf60 con 0x7f16e80f6510 2023-12-08T10:14:29.487 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.485+0000 7f16deffd700 1 -- 172.21.15.170:0/2999930098 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16d801db70 con 0x7f16e80f6510 2023-12-08T10:14:29.487 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:29.487 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f16d402ff60 msgr2=0x7f16d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f16d402ff60 0x7f16d4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f16e40096a0 tx=0x7f16e4006b40).stop 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 msgr2=0x7f16e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f16d8013510 tx=0x7f16d800b040).stop 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 shutdown_connections 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f16d402ff60 0x7f16d4032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:29.489 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 --2- 172.21.15.170:0/2999930098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16e80f6510 0x7f16e8106730 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:29.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 >> 172.21.15.170:0/2999930098 conn(0x7f16e80f1c70 msgr2=0x7f16e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:29.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 shutdown_connections 2023-12-08T10:14:29.490 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:29.486+0000 7f16f0426700 1 -- 172.21.15.170:0/2999930098 wait complete. 2023-12-08T10:14:29.490 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:30.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:28.383109+0000 mgr.smithi023.tgiigg (mgr.14164) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:30.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:30 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:29.486425+0000 mon.smithi023 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.170:0/2999930098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:30.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:30.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:32.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3164464913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 msgr2=0x7fdd600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:32.000 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 --2- 172.21.15.170:0/3164464913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd600f90f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fdd5c009fb0 tx=0x7fdd5c009b30).stop 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3164464913 shutdown_connections 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 --2- 172.21.15.170:0/3164464913 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd600f90f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3164464913 >> 172.21.15.170:0/3164464913 conn(0x7fdd600f0b70 msgr2=0x7fdd600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3164464913 shutdown_connections 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.996+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3164464913 wait complete. 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd67d9a700 1 Processor -- start 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd67d9a700 1 -- start start 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd67d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd601066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd67d9a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd60106bd0 con 0x7fdd60060280 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd65b36700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd601066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd65b36700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd601066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:44468/0 (socket says 172.21.15.170:44468) 2023-12-08T10:14:32.001 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd65b36700 1 -- 172.21.15.170:0/3449265842 learned_addr learned my addr 172.21.15.170:0/3449265842 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.997+0000 7fdd65b36700 1 -- 172.21.15.170:0/3449265842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c005040 con 0x7fdd60060280 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.998+0000 7fdd65b36700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd601066e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fdd5c013510 tx=0x7fdd5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.998+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd5c009320 con 0x7fdd60060280 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.998+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd60106dd0 con 0x7fdd60060280 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.998+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd5c013f00 con 0x7fdd60060280 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.998+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd5c014dc0 con 0x7fdd60060280 2023-12-08T10:14:32.002 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.998+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd60107260 con 0x7fdd60060280 2023-12-08T10:14:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.999+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd5c0095f0 con 0x7fdd60060280 2023-12-08T10:14:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.999+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd60045ad0 con 0x7fdd60060280 2023-12-08T10:14:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.999+0000 7fdd567fc700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd4c02ff60 0x7fdd4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.999+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd5c041000 con 0x7fdd60060280 2023-12-08T10:14:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.999+0000 7fdd65335700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd4c02ff60 0x7fdd4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:31.999+0000 7fdd65335700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd4c02ff60 0x7fdd4c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fdd500096a0 tx=0x7fdd50006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:32.006 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.003+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd5c014420 con 0x7fdd60060280 2023-12-08T10:14:32.212 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.210+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd60108110 con 0x7fdd60060280 2023-12-08T10:14:32.213 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.211+0000 7fdd567fc700 1 -- 172.21.15.170:0/3449265842 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd5c010230 con 0x7fdd60060280 2023-12-08T10:14:32.213 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:32.213 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:32.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd4c02ff60 msgr2=0x7fdd4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:32.214 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd4c02ff60 0x7fdd4c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fdd500096a0 tx=0x7fdd50006b40).stop 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 msgr2=0x7fdd601066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd601066e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fdd5c013510 tx=0x7fdd5c00b040).stop 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 shutdown_connections 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd4c02ff60 0x7fdd4c032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 --2- 172.21.15.170:0/3449265842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd60060280 0x7fdd601066e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 >> 172.21.15.170:0/3449265842 conn(0x7fdd600f0b70 msgr2=0x7fdd600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 shutdown_connections 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:32.212+0000 7fdd67d9a700 1 -- 172.21.15.170:0/3449265842 wait complete. 2023-12-08T10:14:32.215 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:32.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:32 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:30.383473+0000 mgr.smithi023.tgiigg (mgr.14164) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:33.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:32.212181+0000 mon.smithi023 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.170:0/3449265842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:33.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:33 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:32.383898+0000 mgr.smithi023.tgiigg (mgr.14164) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:33.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:33.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:34.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 -- 172.21.15.170:0/3866209636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 msgr2=0x7fb1a40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:34.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 --2- 172.21.15.170:0/3866209636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a40f90f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb198009fb0 tx=0x7fb198009b30).stop 2023-12-08T10:14:34.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 -- 172.21.15.170:0/3866209636 shutdown_connections 2023-12-08T10:14:34.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 --2- 172.21.15.170:0/3866209636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a40f90f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:34.767 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 -- 172.21.15.170:0/3866209636 >> 172.21.15.170:0/3866209636 conn(0x7fb1a40f0b70 msgr2=0x7fb1a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 -- 172.21.15.170:0/3866209636 shutdown_connections 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.763+0000 7fb1a8867700 1 -- 172.21.15.170:0/3866209636 wait complete. 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a8867700 1 Processor -- start 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a8867700 1 -- start start 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a8867700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a41066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a8867700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a4106bb0 con 0x7fb1a4060280 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a41066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a41066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:35402/0 (socket says 172.21.15.170:35402) 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a259c700 1 -- 172.21.15.170:0/3422309367 learned_addr learned my addr 172.21.15.170:0/3422309367 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a259c700 1 -- 172.21.15.170:0/3422309367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb198005040 con 0x7fb1a4060280 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.764+0000 7fb1a259c700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a41066c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb198013510 tx=0x7fb19800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb198009320 con 0x7fb1a4060280 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a4106db0 con 0x7fb1a4060280 2023-12-08T10:14:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb198013f00 con 0x7fb1a4060280 2023-12-08T10:14:34.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb198014dc0 con 0x7fb1a4060280 2023-12-08T10:14:34.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a4107240 con 0x7fb1a4060280 2023-12-08T10:14:34.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb19801d3d0 con 0x7fb1a4060280 2023-12-08T10:14:34.769 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.765+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a4045ad0 con 0x7fb1a4060280 2023-12-08T10:14:34.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.766+0000 7fb192ffd700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb18c02ff60 0x7fb18c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:34.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.766+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1980410e0 con 0x7fb1a4060280 2023-12-08T10:14:34.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.766+0000 7fb1a1d9b700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb18c02ff60 0x7fb18c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:34.770 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.766+0000 7fb1a1d9b700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb18c02ff60 0x7fb18c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fb1940096a0 tx=0x7fb194006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:34.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.770+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1980095f0 con 0x7fb1a4060280 2023-12-08T10:14:34.977 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.975+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1a41080f0 con 0x7fb1a4060280 2023-12-08T10:14:34.978 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.976+0000 7fb192ffd700 1 -- 172.21.15.170:0/3422309367 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb19801cb10 con 0x7fb1a4060280 2023-12-08T10:14:34.978 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:34.978 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:34.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb18c02ff60 msgr2=0x7fb18c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:34.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb18c02ff60 0x7fb18c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fb1940096a0 tx=0x7fb194006b40).stop 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 msgr2=0x7fb1a41066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a41066c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb198013510 tx=0x7fb19800b040).stop 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 shutdown_connections 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb18c02ff60 0x7fb18c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 --2- 172.21.15.170:0/3422309367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1a4060280 0x7fb1a41066c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 >> 172.21.15.170:0/3422309367 conn(0x7fb1a40f0b70 msgr2=0x7fb1a40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 shutdown_connections 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:34.977+0000 7fb1a8867700 1 -- 172.21.15.170:0/3422309367 wait complete. 2023-12-08T10:14:34.980 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:35.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:34.384316+0000 mgr.smithi023.tgiigg (mgr.14164) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:35.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:35 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:34.977412+0000 mon.smithi023 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.170:0/3422309367' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:36.320 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:36.321 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:37.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- 172.21.15.170:0/1959377411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 msgr2=0x7f0ba40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:37.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 --2- 172.21.15.170:0/1959377411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba40f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f0b94009fb0 tx=0x7f0b94009b30).stop 2023-12-08T10:14:37.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- 172.21.15.170:0/1959377411 shutdown_connections 2023-12-08T10:14:37.482 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 --2- 172.21.15.170:0/1959377411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba40f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- 172.21.15.170:0/1959377411 >> 172.21.15.170:0/1959377411 conn(0x7f0ba40f09a0 msgr2=0x7f0ba40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- 172.21.15.170:0/1959377411 shutdown_connections 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- 172.21.15.170:0/1959377411 wait complete. 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 Processor -- start 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- start start 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.479+0000 7f0baa52d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ba4106fc0 con 0x7f0ba40f51a0 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.480+0000 7f0ba952b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.480+0000 7f0ba952b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:35420/0 (socket says 172.21.15.170:35420) 2023-12-08T10:14:37.483 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.480+0000 7f0ba952b700 1 -- 172.21.15.170:0/3593833383 learned_addr learned my addr 172.21.15.170:0/3593833383 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:37.484 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.480+0000 7f0ba952b700 1 -- 172.21.15.170:0/3593833383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b94005040 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.480+0000 7f0ba952b700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba410b5d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f0b94013510 tx=0x7f0b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.480+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b94009320 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.481+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba4060220 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.481+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b94013f00 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.481+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b94014dc0 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.481+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba410bac0 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.482+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b940095f0 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.482+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba4045ad0 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.482+0000 7f0b99ffb700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b9002ff60 0x7f0b90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.482+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b94041040 con 0x7f0ba40f51a0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.482+0000 7f0ba8d2a700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b9002ff60 0x7f0b90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:37.485 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.482+0000 7f0ba8d2a700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b9002ff60 0x7f0b90032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f0ba00096a0 tx=0x7f0ba0006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:37.488 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.486+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b94014420 con 0x7f0ba40f51a0 2023-12-08T10:14:37.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.695+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ba410c280 con 0x7f0ba40f51a0 2023-12-08T10:14:37.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.695+0000 7f0b99ffb700 1 -- 172.21.15.170:0/3593833383 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0b94010230 con 0x7f0ba40f51a0 2023-12-08T10:14:37.698 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:37.698 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:37.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b9002ff60 msgr2=0x7f0b90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:37.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b9002ff60 0x7f0b90032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f0ba00096a0 tx=0x7f0ba0006b40).stop 2023-12-08T10:14:37.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 msgr2=0x7f0ba410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba410b5d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f0b94013510 tx=0x7f0b9400b040).stop 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 shutdown_connections 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f0b9002ff60 0x7f0b90032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 --2- 172.21.15.170:0/3593833383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ba40f51a0 0x7f0ba410b5d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 >> 172.21.15.170:0/3593833383 conn(0x7f0ba40f09a0 msgr2=0x7f0ba40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 shutdown_connections 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:37.697+0000 7f0baa52d700 1 -- 172.21.15.170:0/3593833383 wait complete. 2023-12-08T10:14:37.700 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:37.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:37 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:36.384601+0000 mgr.smithi023.tgiigg (mgr.14164) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:38.932 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:38 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:37.697276+0000 mon.smithi023 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.170:0/3593833383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:39.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:39.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:40.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:39 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:38.384882+0000 mgr.smithi023.tgiigg (mgr.14164) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.218+0000 7f385944a700 1 -- 172.21.15.170:0/111716185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 msgr2=0x7f38540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.218+0000 7f385944a700 1 --2- 172.21.15.170:0/111716185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f38540f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f3844009fb0 tx=0x7f3844009b30).stop 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.218+0000 7f385944a700 1 -- 172.21.15.170:0/111716185 shutdown_connections 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.218+0000 7f385944a700 1 --2- 172.21.15.170:0/111716185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f38540f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.218+0000 7f385944a700 1 -- 172.21.15.170:0/111716185 >> 172.21.15.170:0/111716185 conn(0x7f38540f0b70 msgr2=0x7f38540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.218+0000 7f385944a700 1 -- 172.21.15.170:0/111716185 shutdown_connections 2023-12-08T10:14:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.219+0000 7f385944a700 1 -- 172.21.15.170:0/111716185 wait complete. 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.219+0000 7f385944a700 1 Processor -- start 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.219+0000 7f385944a700 1 -- start start 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.220+0000 7f385944a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f3854106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.220+0000 7f385944a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3854106c20 con 0x7f38540f5410 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.220+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f3854106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.220+0000 7f3852ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f3854106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:35436/0 (socket says 172.21.15.170:35436) 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.220+0000 7f3852ffd700 1 -- 172.21.15.170:0/441779524 learned_addr learned my addr 172.21.15.170:0/441779524 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:40.223 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.220+0000 7f3852ffd700 1 -- 172.21.15.170:0/441779524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3844005040 con 0x7f38540f5410 2023-12-08T10:14:40.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.221+0000 7f3852ffd700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f3854106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3844013510 tx=0x7f384400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:40.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.221+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3844009320 con 0x7f38540f5410 2023-12-08T10:14:40.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.221+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3854106e20 con 0x7f38540f5410 2023-12-08T10:14:40.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.221+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3844013f00 con 0x7f38540f5410 2023-12-08T10:14:40.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.221+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3844014d90 con 0x7f38540f5410 2023-12-08T10:14:40.224 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.221+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38541072b0 con 0x7f38540f5410 2023-12-08T10:14:40.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.222+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3844014420 con 0x7f38540f5410 2023-12-08T10:14:40.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.222+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3854045ad0 con 0x7f38540f5410 2023-12-08T10:14:40.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.222+0000 7f383b7fe700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f383c02ff60 0x7f383c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:40.225 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.222+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3844041000 con 0x7f38540f5410 2023-12-08T10:14:40.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.222+0000 7f38527fc700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f383c02ff60 0x7f383c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:40.226 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.222+0000 7f38527fc700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f383c02ff60 0x7f383c032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f38480096a0 tx=0x7f3848006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:40.229 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.227+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38440095f0 con 0x7f38540f5410 2023-12-08T10:14:40.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.433+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3854108160 con 0x7f38540f5410 2023-12-08T10:14:40.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.434+0000 7f383b7fe700 1 -- 172.21.15.170:0/441779524 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f384401db70 con 0x7f38540f5410 2023-12-08T10:14:40.436 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:40.436 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f383c02ff60 msgr2=0x7f383c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f383c02ff60 0x7f383c032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f38480096a0 tx=0x7f3848006b40).stop 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 msgr2=0x7f3854106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f3854106730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3844013510 tx=0x7f384400b040).stop 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 shutdown_connections 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f383c02ff60 0x7f383c032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 --2- 172.21.15.170:0/441779524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38540f5410 0x7f3854106730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 >> 172.21.15.170:0/441779524 conn(0x7f38540f0b70 msgr2=0x7f38540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 shutdown_connections 2023-12-08T10:14:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:40.435+0000 7f385944a700 1 -- 172.21.15.170:0/441779524 wait complete. 2023-12-08T10:14:40.439 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:41.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:40 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:40.435517+0000 mon.smithi023 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.170:0/441779524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:41.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:41.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:42.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:41 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:40.385187+0000 mgr.smithi023.tgiigg (mgr.14164) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:42.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 -- 172.21.15.170:0/680243943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 msgr2=0x7f06440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:42.973 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 --2- 172.21.15.170:0/680243943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06440f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f0634009fb0 tx=0x7f0634009b30).stop 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 -- 172.21.15.170:0/680243943 shutdown_connections 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 --2- 172.21.15.170:0/680243943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06440f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 -- 172.21.15.170:0/680243943 >> 172.21.15.170:0/680243943 conn(0x7f06440f0b70 msgr2=0x7f06440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 -- 172.21.15.170:0/680243943 shutdown_connections 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.969+0000 7f064c3f9700 1 -- 172.21.15.170:0/680243943 wait complete. 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064c3f9700 1 Processor -- start 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064c3f9700 1 -- start start 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064c3f9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06441093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064c3f9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0644060280 con 0x7f06440f5410 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064a195700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06441093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064a195700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06441093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:42386/0 (socket says 172.21.15.170:42386) 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.970+0000 7f064a195700 1 -- 172.21.15.170:0/4077782888 learned_addr learned my addr 172.21.15.170:0/4077782888 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:42.974 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f064a195700 1 -- 172.21.15.170:0/4077782888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0634005040 con 0x7f06440f5410 2023-12-08T10:14:42.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f064a195700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06441093f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0634013510 tx=0x7f063400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:42.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0634009320 con 0x7f06440f5410 2023-12-08T10:14:42.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0644060480 con 0x7f06440f5410 2023-12-08T10:14:42.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0634013f00 con 0x7f06440f5410 2023-12-08T10:14:42.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0634014d90 con 0x7f06440f5410 2023-12-08T10:14:42.975 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.971+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0644109b20 con 0x7f06440f5410 2023-12-08T10:14:42.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.972+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06340263a0 con 0x7f06440f5410 2023-12-08T10:14:42.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.972+0000 7f063affd700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f063002ff60 0x7f0630032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:42.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.972+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0634041310 con 0x7f06440f5410 2023-12-08T10:14:42.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.972+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0644045ad0 con 0x7f06440f5410 2023-12-08T10:14:42.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.972+0000 7f0649994700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f063002ff60 0x7f0630032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:42.976 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.973+0000 7f0649994700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f063002ff60 0x7f0630032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f06400096a0 tx=0x7f0640006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:42.979 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:42.977+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0634012920 con 0x7f06440f5410 2023-12-08T10:14:43.186 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.184+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f064410a240 con 0x7f06440f5410 2023-12-08T10:14:43.187 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.185+0000 7f063affd700 1 -- 172.21.15.170:0/4077782888 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f063402b030 con 0x7f06440f5410 2023-12-08T10:14:43.187 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:43.187 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f063002ff60 msgr2=0x7f0630032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f063002ff60 0x7f0630032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f06400096a0 tx=0x7f0640006b40).stop 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 msgr2=0x7f06441093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06441093f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f0634013510 tx=0x7f063400b040).stop 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 shutdown_connections 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f063002ff60 0x7f0630032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 --2- 172.21.15.170:0/4077782888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440f5410 0x7f06441093f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 >> 172.21.15.170:0/4077782888 conn(0x7f06440f0b70 msgr2=0x7f06440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 shutdown_connections 2023-12-08T10:14:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:43.186+0000 7f064c3f9700 1 -- 172.21.15.170:0/4077782888 wait complete. 2023-12-08T10:14:43.190 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:42.385479+0000 mgr.smithi023.tgiigg (mgr.14164) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:44.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:43 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:43.186271+0000 mon.smithi023 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.170:0/4077782888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:44.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:44.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.693+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2363680205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 msgr2=0x7f4da40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.693+0000 7f4da9f6d700 1 --2- 172.21.15.170:0/2363680205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f90f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f4d94009fb0 tx=0x7f4d94009b30).stop 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2363680205 shutdown_connections 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 --2- 172.21.15.170:0/2363680205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f90f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2363680205 >> 172.21.15.170:0/2363680205 conn(0x7f4da40f0b70 msgr2=0x7f4da40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2363680205 shutdown_connections 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2363680205 wait complete. 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 Processor -- start 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 -- start start 2023-12-08T10:14:45.697 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:45.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.694+0000 7f4da9f6d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da40f4ff0 con 0x7f4da4060280 2023-12-08T10:14:45.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:45.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:42398/0 (socket says 172.21.15.170:42398) 2023-12-08T10:14:45.698 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da37fe700 1 -- 172.21.15.170:0/2811043153 learned_addr learned my addr 172.21.15.170:0/2811043153 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:45.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da37fe700 1 -- 172.21.15.170:0/2811043153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d94005040 con 0x7f4da4060280 2023-12-08T10:14:45.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da37fe700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f6b70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4d94013510 tx=0x7f4d9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:45.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d94009320 con 0x7f4da4060280 2023-12-08T10:14:45.699 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da40f5190 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d94013f00 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d94014dc0 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.695+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da40f5620 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.696+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d9401d3d0 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.696+0000 7f4d9bfff700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d9002ffb0 0x7f4d90032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.696+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d94041120 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.696+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da4059100 con 0x7f4da4060280 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.697+0000 7f4da2ffd700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d9002ffb0 0x7f4d90032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:45.700 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.697+0000 7f4da2ffd700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d9002ffb0 0x7f4d90032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4d8c0096a0 tx=0x7f4d8c006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:45.703 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.701+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d94026490 con 0x7f4da4060280 2023-12-08T10:14:45.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.908+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4da4045ad0 con 0x7f4da4060280 2023-12-08T10:14:45.911 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.909+0000 7f4d9bfff700 1 -- 172.21.15.170:0/2811043153 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4d940126f0 con 0x7f4da4060280 2023-12-08T10:14:45.911 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:45.912 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.910+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d9002ffb0 msgr2=0x7f4d90032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.910+0000 7f4da9f6d700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d9002ffb0 0x7f4d90032470 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4d8c0096a0 tx=0x7f4d8c006b40).stop 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.910+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 msgr2=0x7f4da40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.910+0000 7f4da9f6d700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f6b70 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4d94013510 tx=0x7f4d9400b040).stop 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.911+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 shutdown_connections 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.911+0000 7f4da9f6d700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f4d9002ffb0 0x7f4d90032470 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.911+0000 7f4da9f6d700 1 --2- 172.21.15.170:0/2811043153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4da4060280 0x7f4da40f6b70 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.911+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 >> 172.21.15.170:0/2811043153 conn(0x7f4da40f0b70 msgr2=0x7f4da40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.911+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 shutdown_connections 2023-12-08T10:14:45.913 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:45.911+0000 7f4da9f6d700 1 -- 172.21.15.170:0/2811043153 wait complete. 2023-12-08T10:14:45.914 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:46.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:45 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:44.385836+0000 mgr.smithi023.tgiigg (mgr.14164) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:47.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:46 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:45.910812+0000 mon.smithi023 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.170:0/2811043153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:47.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:47.294 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:48.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:47 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:46.386136+0000 mgr.smithi023.tgiigg (mgr.14164) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:48.435 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 -- 172.21.15.170:0/1299955796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 msgr2=0x7f18f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:48.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 --2- 172.21.15.170:0/1299955796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f80f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f18ec009fb0 tx=0x7f18ec009b30).stop 2023-12-08T10:14:48.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 -- 172.21.15.170:0/1299955796 shutdown_connections 2023-12-08T10:14:48.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 --2- 172.21.15.170:0/1299955796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f80f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:48.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 -- 172.21.15.170:0/1299955796 >> 172.21.15.170:0/1299955796 conn(0x7f18f80f0b70 msgr2=0x7f18f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:48.436 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 -- 172.21.15.170:0/1299955796 shutdown_connections 2023-12-08T10:14:48.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.428+0000 7f18fe079700 1 -- 172.21.15.170:0/1299955796 wait complete. 2023-12-08T10:14:48.437 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.429+0000 7f18fe079700 1 Processor -- start 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.429+0000 7f18fe079700 1 -- start start 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.429+0000 7f18fe079700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.429+0000 7f18fe079700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8105b20 con 0x7f18f80f5410 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18f77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18f77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:42414/0 (socket says 172.21.15.170:42414) 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18f77fe700 1 -- 172.21.15.170:0/1212620113 learned_addr learned my addr 172.21.15.170:0/1212620113 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18f77fe700 1 -- 172.21.15.170:0/1212620113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec005040 con 0x7f18f80f5410 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18f77fe700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f8105630 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f18ec013510 tx=0x7f18ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18ec009320 con 0x7f18f80f5410 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f8105d20 con 0x7f18f80f5410 2023-12-08T10:14:48.438 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18ec013f00 con 0x7f18f80f5410 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18ec014d90 con 0x7f18f80f5410 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.430+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f81061b0 con 0x7f18f80f5410 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.431+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18ec0095f0 con 0x7f18f80f5410 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.431+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f8045ad0 con 0x7f18f80f5410 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.431+0000 7f18dffff700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f18e002ff60 0x7f18e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.432+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18ec040fe0 con 0x7f18f80f5410 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.432+0000 7f18f6ffd700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f18e002ff60 0x7f18e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:48.439 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.432+0000 7f18f6ffd700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f18e002ff60 0x7f18e0032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f18e80096a0 tx=0x7f18e8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:48.440 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.437+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18ec01d7f0 con 0x7f18f80f5410 2023-12-08T10:14:48.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.646+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18f80fae60 con 0x7f18f80f5410 2023-12-08T10:14:48.648 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.647+0000 7f18dffff700 1 -- 172.21.15.170:0/1212620113 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18ec010230 con 0x7f18f80f5410 2023-12-08T10:14:48.649 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:48.649 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:48.650 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.648+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f18e002ff60 msgr2=0x7f18e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.648+0000 7f18fe079700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f18e002ff60 0x7f18e0032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f18e80096a0 tx=0x7f18e8006b40).stop 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 msgr2=0x7f18f8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f8105630 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f18ec013510 tx=0x7f18ec00b040).stop 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 shutdown_connections 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f18e002ff60 0x7f18e0032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 --2- 172.21.15.170:0/1212620113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18f80f5410 0x7f18f8105630 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 >> 172.21.15.170:0/1212620113 conn(0x7f18f80f0b70 msgr2=0x7f18f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 shutdown_connections 2023-12-08T10:14:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:48.649+0000 7f18fe079700 1 -- 172.21.15.170:0/1212620113 wait complete. 2023-12-08T10:14:48.652 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:49.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:48 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:48.647851+0000 mon.smithi023 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.170:0/1212620113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:50.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:50.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:50.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:49 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:48.386531+0000 mgr.smithi023.tgiigg (mgr.14164) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:51.171 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 -- 172.21.15.170:0/590580110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 msgr2=0x7fb1600f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 --2- 172.21.15.170:0/590580110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1600f6e30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb154009fb0 tx=0x7fb154009b30).stop 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 -- 172.21.15.170:0/590580110 shutdown_connections 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 --2- 172.21.15.170:0/590580110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1600f6e30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 -- 172.21.15.170:0/590580110 >> 172.21.15.170:0/590580110 conn(0x7fb1600f02e0 msgr2=0x7fb1600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 -- 172.21.15.170:0/590580110 shutdown_connections 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.168+0000 7fb165766700 1 -- 172.21.15.170:0/590580110 wait complete. 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb165766700 1 Processor -- start 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb165766700 1 -- start start 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb165766700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1601093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb165766700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1600f5350 con 0x7fb1600f6a10 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb15effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1601093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb15effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1601093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:42432/0 (socket says 172.21.15.170:42432) 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.169+0000 7fb15effd700 1 -- 172.21.15.170:0/3741936460 learned_addr learned my addr 172.21.15.170:0/3741936460 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:51.172 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb15effd700 1 -- 172.21.15.170:0/3741936460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154005040 con 0x7fb1600f6a10 2023-12-08T10:14:51.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb15effd700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1601093d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fb154013510 tx=0x7fb15400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:51.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb154009320 con 0x7fb1600f6a10 2023-12-08T10:14:51.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1600f5550 con 0x7fb1600f6a10 2023-12-08T10:14:51.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb154013f00 con 0x7fb1600f6a10 2023-12-08T10:14:51.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb154014d90 con 0x7fb1600f6a10 2023-12-08T10:14:51.173 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.170+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb160109a80 con 0x7fb1600f6a10 2023-12-08T10:14:51.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.171+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb160045ad0 con 0x7fb1600f6a10 2023-12-08T10:14:51.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.171+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1540095f0 con 0x7fb1600f6a10 2023-12-08T10:14:51.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.171+0000 7fb1477fe700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb14802ff60 0x7fb148032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:51.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.171+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb154041000 con 0x7fb1600f6a10 2023-12-08T10:14:51.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.171+0000 7fb15e7fc700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb14802ff60 0x7fb148032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:51.174 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.172+0000 7fb15e7fc700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb14802ff60 0x7fb148032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb1500096a0 tx=0x7fb150006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:51.177 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.176+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb154014420 con 0x7fb1600f6a10 2023-12-08T10:14:51.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.384+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb16010a2a0 con 0x7fb1600f6a10 2023-12-08T10:14:51.386 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.385+0000 7fb1477fe700 1 -- 172.21.15.170:0/3741936460 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb154010230 con 0x7fb1600f6a10 2023-12-08T10:14:51.386 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:51.386 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb14802ff60 msgr2=0x7fb148032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb14802ff60 0x7fb148032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fb1500096a0 tx=0x7fb150006b40).stop 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 msgr2=0x7fb1601093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1601093d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fb154013510 tx=0x7fb15400b040).stop 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 shutdown_connections 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fb14802ff60 0x7fb148032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 --2- 172.21.15.170:0/3741936460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1600f6a10 0x7fb1601093d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.386+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 >> 172.21.15.170:0/3741936460 conn(0x7fb1600f02e0 msgr2=0x7fb1600f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.387+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 shutdown_connections 2023-12-08T10:14:51.388 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:51.387+0000 7fb165766700 1 -- 172.21.15.170:0/3741936460 wait complete. 2023-12-08T10:14:51.389 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:50.386840+0000 mgr.smithi023.tgiigg (mgr.14164) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:52.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:51 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:51.385607+0000 mon.smithi023 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.170:0/3741936460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:52.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:52.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:53.879 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 -- 172.21.15.170:0/619124160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 msgr2=0x7ffa6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 --2- 172.21.15.170:0/619124160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c0f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7ffa5c009fb0 tx=0x7ffa5c009b30).stop 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 -- 172.21.15.170:0/619124160 shutdown_connections 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 --2- 172.21.15.170:0/619124160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c0f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 -- 172.21.15.170:0/619124160 >> 172.21.15.170:0/619124160 conn(0x7ffa6c0f0b70 msgr2=0x7ffa6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 -- 172.21.15.170:0/619124160 shutdown_connections 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.877+0000 7ffa7463b700 1 -- 172.21.15.170:0/619124160 wait complete. 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.878+0000 7ffa7463b700 1 Processor -- start 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.878+0000 7ffa7463b700 1 -- start start 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.878+0000 7ffa7463b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:53.880 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.878+0000 7ffa7463b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c060240 con 0x7ffa6c0f5410 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa723d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa723d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:57960/0 (socket says 172.21.15.170:57960) 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa723d7700 1 -- 172.21.15.170:0/3377261880 learned_addr learned my addr 172.21.15.170:0/3377261880 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa723d7700 1 -- 172.21.15.170:0/3377261880 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c005040 con 0x7ffa6c0f5410 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa723d7700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c109430 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ffa5c013510 tx=0x7ffa5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa5c009320 con 0x7ffa6c0f5410 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c060440 con 0x7ffa6c0f5410 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa5c013f00 con 0x7ffa6c0f5410 2023-12-08T10:14:53.882 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.879+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa5c014d90 con 0x7ffa6c0f5410 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.880+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c109ae0 con 0x7ffa6c0f5410 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.880+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa5c0095f0 con 0x7ffa6c0f5410 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.880+0000 7ffa62ffd700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ffa5802ff60 0x7ffa58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.880+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa5c041000 con 0x7ffa6c0f5410 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.880+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c045ad0 con 0x7ffa6c0f5410 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.881+0000 7ffa71bd6700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ffa5802ff60 0x7ffa58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:53.883 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.881+0000 7ffa71bd6700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ffa5802ff60 0x7ffa58032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ffa680096a0 tx=0x7ffa68006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:53.886 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:53.885+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa5c014420 con 0x7ffa6c0f5410 2023-12-08T10:14:54.093 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.092+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa6c0faf50 con 0x7ffa6c0f5410 2023-12-08T10:14:54.094 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.092+0000 7ffa62ffd700 1 -- 172.21.15.170:0/3377261880 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa5c02b020 con 0x7ffa6c0f5410 2023-12-08T10:14:54.094 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:54.094 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ffa5802ff60 msgr2=0x7ffa58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ffa5802ff60 0x7ffa58032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7ffa680096a0 tx=0x7ffa68006b40).stop 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 msgr2=0x7ffa6c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c109430 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7ffa5c013510 tx=0x7ffa5c00b040).stop 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 shutdown_connections 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7ffa5802ff60 0x7ffa58032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 --2- 172.21.15.170:0/3377261880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa6c0f5410 0x7ffa6c109430 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 >> 172.21.15.170:0/3377261880 conn(0x7ffa6c0f0b70 msgr2=0x7ffa6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 shutdown_connections 2023-12-08T10:14:54.096 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:54.094+0000 7ffa7463b700 1 -- 172.21.15.170:0/3377261880 wait complete. 2023-12-08T10:14:54.097 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:54.182 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:53 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:52.387191+0000 mgr.smithi023.tgiigg (mgr.14164) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:55.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:54 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:54.093262+0000 mon.smithi023 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.170:0/3377261880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:55.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:55.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:56.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:55 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:54.387507+0000 mgr.smithi023.tgiigg (mgr.14164) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:56.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 -- 172.21.15.170:0/1990935553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 msgr2=0x7f9bcc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:56.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 --2- 172.21.15.170:0/1990935553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc0f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9bbc009fb0 tx=0x7f9bbc009b30).stop 2023-12-08T10:14:56.610 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 -- 172.21.15.170:0/1990935553 shutdown_connections 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 --2- 172.21.15.170:0/1990935553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc0f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 -- 172.21.15.170:0/1990935553 >> 172.21.15.170:0/1990935553 conn(0x7f9bcc0f1c70 msgr2=0x7f9bcc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 -- 172.21.15.170:0/1990935553 shutdown_connections 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.607+0000 7f9bd37e1700 1 -- 172.21.15.170:0/1990935553 wait complete. 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd37e1700 1 Processor -- start 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd37e1700 1 -- start start 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd37e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd37e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc106c20 con 0x7f9bcc0f6510 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd157d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd157d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:57978/0 (socket says 172.21.15.170:57978) 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd157d700 1 -- 172.21.15.170:0/2946229336 learned_addr learned my addr 172.21.15.170:0/2946229336 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd157d700 1 -- 172.21.15.170:0/2946229336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc005040 con 0x7f9bcc0f6510 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.608+0000 7f9bd157d700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9bbc014000 tx=0x7f9bbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:56.611 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.609+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bbc0095f0 con 0x7f9bcc0f6510 2023-12-08T10:14:56.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.609+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc106e20 con 0x7f9bcc0f6510 2023-12-08T10:14:56.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.609+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bbc007e70 con 0x7f9bcc0f6510 2023-12-08T10:14:56.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.609+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bbc014ea0 con 0x7f9bcc0f6510 2023-12-08T10:14:56.612 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.609+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc1072b0 con 0x7f9bcc0f6510 2023-12-08T10:14:56.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.610+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bbc025070 con 0x7f9bcc0f6510 2023-12-08T10:14:56.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.610+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc045ad0 con 0x7f9bcc0f6510 2023-12-08T10:14:56.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.610+0000 7f9bc1ffb700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bb802ff60 0x7f9bb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:56.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.610+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bbc040940 con 0x7f9bcc0f6510 2023-12-08T10:14:56.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.610+0000 7f9bd0d7c700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bb802ff60 0x7f9bb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:56.613 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.611+0000 7f9bd0d7c700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bb802ff60 0x7f9bb8032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9bc80096a0 tx=0x7f9bc8006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:56.615 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.614+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bbc0079c0 con 0x7f9bcc0f6510 2023-12-08T10:14:56.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.823+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bcc0fbf60 con 0x7f9bcc0f6510 2023-12-08T10:14:56.825 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.824+0000 7f9bc1ffb700 1 -- 172.21.15.170:0/2946229336 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9bbc00f300 con 0x7f9bcc0f6510 2023-12-08T10:14:56.826 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:56.826 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:56.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bb802ff60 msgr2=0x7f9bb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:56.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bb802ff60 0x7f9bb8032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f9bc80096a0 tx=0x7f9bc8006b40).stop 2023-12-08T10:14:56.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 msgr2=0x7f9bcc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:56.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9bbc014000 tx=0x7f9bbc00b040).stop 2023-12-08T10:14:56.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 shutdown_connections 2023-12-08T10:14:56.827 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9bb802ff60 0x7f9bb8032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:56.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 --2- 172.21.15.170:0/2946229336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bcc0f6510 0x7f9bcc106730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:56.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 >> 172.21.15.170:0/2946229336 conn(0x7f9bcc0f1c70 msgr2=0x7f9bcc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:56.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 shutdown_connections 2023-12-08T10:14:56.828 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:56.826+0000 7f9bd37e1700 1 -- 172.21.15.170:0/2946229336 wait complete. 2023-12-08T10:14:56.828 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:14:57.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:56 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:56.824904+0000 mon.smithi023 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.170:0/2946229336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:14:58.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:14:58.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:14:58.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:57 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:56.387835+0000 mgr.smithi023.tgiigg (mgr.14164) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:14:59.343 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 -- 172.21.15.170:0/3930308199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 msgr2=0x7f18140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 --2- 172.21.15.170:0/3930308199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f18140f6930 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f180c009960 tx=0x7f180c004d10).stop 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 -- 172.21.15.170:0/3930308199 shutdown_connections 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 --2- 172.21.15.170:0/3930308199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f18140f6930 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 -- 172.21.15.170:0/3930308199 >> 172.21.15.170:0/3930308199 conn(0x7f18140f1c70 msgr2=0x7f18140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 -- 172.21.15.170:0/3930308199 shutdown_connections 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.340+0000 7f181b0aa700 1 -- 172.21.15.170:0/3930308199 wait complete. 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f181b0aa700 1 Processor -- start 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f181b0aa700 1 -- start start 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f181b0aa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f1814106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f181b0aa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1814106c20 con 0x7f18140f6510 2023-12-08T10:14:59.344 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f1818e46700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f1814106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f1818e46700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f1814106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:58010/0 (socket says 172.21.15.170:58010) 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.341+0000 7f1818e46700 1 -- 172.21.15.170:0/972168566 learned_addr learned my addr 172.21.15.170:0/972168566 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f1818e46700 1 -- 172.21.15.170:0/972168566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c009640 con 0x7f18140f6510 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f1818e46700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f1814106730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f180c004b70 tx=0x7f180c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f180c018070 con 0x7f18140f6510 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1814106e20 con 0x7f18140f6510 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f180c00b870 con 0x7f18140f6510 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f180c01c560 con 0x7f18140f6510 2023-12-08T10:14:59.345 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.342+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18141072b0 con 0x7f18140f6510 2023-12-08T10:14:59.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.343+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f180c01c6c0 con 0x7f18140f6510 2023-12-08T10:14:59.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.343+0000 7f18117fa700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f17fc02ff60 0x7f17fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:14:59.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.343+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f180c041220 con 0x7f18140f6510 2023-12-08T10:14:59.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.343+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1814045ad0 con 0x7f18140f6510 2023-12-08T10:14:59.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.343+0000 7f1813fff700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f17fc02ff60 0x7f17fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:14:59.346 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.344+0000 7f1813fff700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f17fc02ff60 0x7f17fc032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f18080096a0 tx=0x7f1808006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:14:59.349 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.348+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f180c0127f0 con 0x7f18140f6510 2023-12-08T10:14:59.554 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.553+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18140fbf60 con 0x7f18140f6510 2023-12-08T10:14:59.555 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.554+0000 7f18117fa700 1 -- 172.21.15.170:0/972168566 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f180c019230 con 0x7f18140f6510 2023-12-08T10:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:14:59.555 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.555+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f17fc02ff60 msgr2=0x7f17fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.555+0000 7f181b0aa700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f17fc02ff60 0x7f17fc032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f18080096a0 tx=0x7f1808006b40).stop 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.555+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 msgr2=0x7f1814106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.555+0000 7f181b0aa700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f1814106730 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f180c004b70 tx=0x7f180c004d10).stop 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.556+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 shutdown_connections 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.556+0000 7f181b0aa700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f17fc02ff60 0x7f17fc032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.556+0000 7f181b0aa700 1 --2- 172.21.15.170:0/972168566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18140f6510 0x7f1814106730 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.556+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 >> 172.21.15.170:0/972168566 conn(0x7f18140f1c70 msgr2=0x7f18140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:14:59.558 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.556+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 shutdown_connections 2023-12-08T10:14:59.559 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:14:59.556+0000 7f181b0aa700 1 -- 172.21.15.170:0/972168566 wait complete. 2023-12-08T10:14:59.559 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:14:58.388067+0000 mgr.smithi023.tgiigg (mgr.14164) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:00.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:14:59 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:14:59.554468+0000 mon.smithi023 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.170:0/972168566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:00.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:00.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:02.041 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3984708901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 msgr2=0x7f6e4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 --2- 172.21.15.170:0/3984708901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c0f5830 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f6e3c009fb0 tx=0x7f6e3c009b30).stop 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3984708901 shutdown_connections 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 --2- 172.21.15.170:0/3984708901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c0f5830 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3984708901 >> 172.21.15.170:0/3984708901 conn(0x7f6e4c0f0b70 msgr2=0x7f6e4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3984708901 shutdown_connections 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.038+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3984708901 wait complete. 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.039+0000 7f6e50e1a700 1 Processor -- start 2023-12-08T10:15:02.042 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.039+0000 7f6e50e1a700 1 -- start start 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.039+0000 7f6e50e1a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.039+0000 7f6e50e1a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c060240 con 0x7f6e4c0f5410 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:58026/0 (socket says 172.21.15.170:58026) 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e4a59c700 1 -- 172.21.15.170:0/3659923115 learned_addr learned my addr 172.21.15.170:0/3659923115 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e4a59c700 1 -- 172.21.15.170:0/3659923115 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c005040 con 0x7f6e4c0f5410 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e4a59c700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c109430 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f6e3c013510 tx=0x7f6e3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:02.043 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e3c009320 con 0x7f6e4c0f5410 2023-12-08T10:15:02.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c060440 con 0x7f6e4c0f5410 2023-12-08T10:15:02.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6e3c013f00 con 0x7f6e4c0f5410 2023-12-08T10:15:02.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e3c014d90 con 0x7f6e4c0f5410 2023-12-08T10:15:02.044 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.040+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c109ae0 con 0x7f6e4c0f5410 2023-12-08T10:15:02.045 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.041+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e3c0095f0 con 0x7f6e4c0f5410 2023-12-08T10:15:02.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.041+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c045ad0 con 0x7f6e4c0f5410 2023-12-08T10:15:02.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.041+0000 7f6e3affd700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e3402ff60 0x7f6e34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:02.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.041+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e3c041000 con 0x7f6e4c0f5410 2023-12-08T10:15:02.046 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.042+0000 7f6e49d9b700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e3402ff60 0x7f6e34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:02.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.042+0000 7f6e49d9b700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e3402ff60 0x7f6e34032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6e400096a0 tx=0x7f6e40006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:02.047 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.046+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e3c014420 con 0x7f6e4c0f5410 2023-12-08T10:15:02.253 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.252+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e4c0fae60 con 0x7f6e4c0f5410 2023-12-08T10:15:02.254 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.253+0000 7f6e3affd700 1 -- 172.21.15.170:0/3659923115 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e3c01d7f0 con 0x7f6e4c0f5410 2023-12-08T10:15:02.254 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:02.254 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e3402ff60 msgr2=0x7f6e34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e3402ff60 0x7f6e34032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f6e400096a0 tx=0x7f6e40006b40).stop 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 msgr2=0x7f6e4c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c109430 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f6e3c013510 tx=0x7f6e3c00b040).stop 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 shutdown_connections 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f6e3402ff60 0x7f6e34032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 --2- 172.21.15.170:0/3659923115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e4c0f5410 0x7f6e4c109430 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 >> 172.21.15.170:0/3659923115 conn(0x7f6e4c0f0b70 msgr2=0x7f6e4c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 shutdown_connections 2023-12-08T10:15:02.256 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:02.254+0000 7f6e50e1a700 1 -- 172.21.15.170:0/3659923115 wait complete. 2023-12-08T10:15:02.257 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:02.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:01 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:00.388427+0000 mgr.smithi023.tgiigg (mgr.14164) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:03.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:02 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:02.253301+0000 mon.smithi023 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.170:0/3659923115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:03.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:03.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:04.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:03 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:02.388764+0000 mgr.smithi023.tgiigg (mgr.14164) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:04.771 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 -- 172.21.15.170:0/4026870057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 msgr2=0x7f04780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 --2- 172.21.15.170:0/4026870057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f04780f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f0468009fb0 tx=0x7f0468009b30).stop 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 -- 172.21.15.170:0/4026870057 shutdown_connections 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 --2- 172.21.15.170:0/4026870057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f04780f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 -- 172.21.15.170:0/4026870057 >> 172.21.15.170:0/4026870057 conn(0x7f04780f0b70 msgr2=0x7f04780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 -- 172.21.15.170:0/4026870057 shutdown_connections 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.768+0000 7f047f931700 1 -- 172.21.15.170:0/4026870057 wait complete. 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047f931700 1 Processor -- start 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047f931700 1 -- start start 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047f931700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f047810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047f931700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0478107240 con 0x7f04780f5410 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047d6cd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f047810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047d6cd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f047810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59230/0 (socket says 172.21.15.170:59230) 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.769+0000 7f047d6cd700 1 -- 172.21.15.170:0/598165057 learned_addr learned my addr 172.21.15.170:0/598165057 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:04.772 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f047d6cd700 1 -- 172.21.15.170:0/598165057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468005040 con 0x7f04780f5410 2023-12-08T10:15:04.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f047d6cd700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f047810b850 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f0468013510 tx=0x7f046800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:04.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0468009320 con 0x7f04780f5410 2023-12-08T10:15:04.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0478060220 con 0x7f04780f5410 2023-12-08T10:15:04.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0468013f00 con 0x7f04780f5410 2023-12-08T10:15:04.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0468014d90 con 0x7f04780f5410 2023-12-08T10:15:04.773 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.770+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047810bd40 con 0x7f04780f5410 2023-12-08T10:15:04.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.771+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04680263a0 con 0x7f04780f5410 2023-12-08T10:15:04.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.771+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0478045ad0 con 0x7f04780f5410 2023-12-08T10:15:04.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.771+0000 7f046dffb700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f046402ff60 0x7f0464032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:04.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.771+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0468040ff0 con 0x7f04780f5410 2023-12-08T10:15:04.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.771+0000 7f047cecc700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f046402ff60 0x7f0464032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:04.774 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.772+0000 7f047cecc700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f046402ff60 0x7f0464032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f04740096a0 tx=0x7f0474006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:04.777 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.776+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04680095f0 con 0x7f04780f5410 2023-12-08T10:15:04.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.984+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04780fae60 con 0x7f04780f5410 2023-12-08T10:15:04.985 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.984+0000 7f046dffb700 1 -- 172.21.15.170:0/598165057 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f046801db70 con 0x7f04780f5410 2023-12-08T10:15:04.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:04.986 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:04.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f046402ff60 msgr2=0x7f0464032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:04.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f046402ff60 0x7f0464032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f04740096a0 tx=0x7f0474006b40).stop 2023-12-08T10:15:04.988 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 msgr2=0x7f047810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f047810b850 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f0468013510 tx=0x7f046800b040).stop 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 shutdown_connections 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f046402ff60 0x7f0464032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 --2- 172.21.15.170:0/598165057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04780f5410 0x7f047810b850 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 >> 172.21.15.170:0/598165057 conn(0x7f04780f0b70 msgr2=0x7f04780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 shutdown_connections 2023-12-08T10:15:04.989 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:04.986+0000 7f047f931700 1 -- 172.21.15.170:0/598165057 wait complete. 2023-12-08T10:15:04.990 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:06.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:06.385 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:04.389115+0000 mgr.smithi023.tgiigg (mgr.14164) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:06.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:05 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:04.985223+0000 mon.smithi023 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.170:0/598165057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:07.525 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/3580922098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 msgr2=0x7f9c380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 --2- 172.21.15.170:0/3580922098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c380f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9c2c009fb0 tx=0x7f9c2c009b30).stop 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/3580922098 shutdown_connections 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 --2- 172.21.15.170:0/3580922098 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c380f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/3580922098 >> 172.21.15.170:0/3580922098 conn(0x7f9c380f1c70 msgr2=0x7f9c380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/3580922098 shutdown_connections 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/3580922098 wait complete. 2023-12-08T10:15:07.526 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 Processor -- start 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.521+0000 7f9c3f1b9700 1 -- start start 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3f1b9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3f1b9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c38106c20 con 0x7f9c380f6510 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3cf55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3cf55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59262/0 (socket says 172.21.15.170:59262) 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3cf55700 1 -- 172.21.15.170:0/650571168 learned_addr learned my addr 172.21.15.170:0/650571168 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3cf55700 1 -- 172.21.15.170:0/650571168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c005040 con 0x7f9c380f6510 2023-12-08T10:15:07.527 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c3cf55700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c38106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9c2c013510 tx=0x7f9c2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:07.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.522+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c2c009320 con 0x7f9c380f6510 2023-12-08T10:15:07.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.523+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c38106e20 con 0x7f9c380f6510 2023-12-08T10:15:07.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.523+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c2c013f00 con 0x7f9c380f6510 2023-12-08T10:15:07.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.523+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c2c014d90 con 0x7f9c380f6510 2023-12-08T10:15:07.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.523+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c381072b0 con 0x7f9c380f6510 2023-12-08T10:15:07.528 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.523+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c2c0095f0 con 0x7f9c380f6510 2023-12-08T10:15:07.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.523+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c38045ad0 con 0x7f9c380f6510 2023-12-08T10:15:07.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.524+0000 7f9c357fa700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9c2802ff60 0x7f9c28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:07.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.524+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c2c040fc0 con 0x7f9c380f6510 2023-12-08T10:15:07.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.524+0000 7f9c37fff700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9c2802ff60 0x7f9c28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:07.529 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.524+0000 7f9c37fff700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9c2802ff60 0x7f9c28032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9c240096a0 tx=0x7f9c24006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:07.530 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.528+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c2c0268c0 con 0x7f9c380f6510 2023-12-08T10:15:07.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.732+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c380fbf60 con 0x7f9c380f6510 2023-12-08T10:15:07.734 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.733+0000 7f9c357fa700 1 -- 172.21.15.170:0/650571168 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c2c010230 con 0x7f9c380f6510 2023-12-08T10:15:07.735 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:07.735 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9c2802ff60 msgr2=0x7f9c28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9c2802ff60 0x7f9c28032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9c240096a0 tx=0x7f9c24006b40).stop 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 msgr2=0x7f9c38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c38106730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9c2c013510 tx=0x7f9c2c00b040).stop 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 shutdown_connections 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f9c2802ff60 0x7f9c28032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 --2- 172.21.15.170:0/650571168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c380f6510 0x7f9c38106730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 >> 172.21.15.170:0/650571168 conn(0x7f9c380f1c70 msgr2=0x7f9c380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:07.736 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 shutdown_connections 2023-12-08T10:15:07.737 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:07.735+0000 7f9c3f1b9700 1 -- 172.21.15.170:0/650571168 wait complete. 2023-12-08T10:15:07.737 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:06.389529+0000 mgr.smithi023.tgiigg (mgr.14164) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:08.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:07 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:07.733925+0000 mon.smithi023 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.170:0/650571168' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:09.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:09.118 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.256+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1266138910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 msgr2=0x7fd1980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.256+0000 7fd19cdd6700 1 --2- 172.21.15.170:0/1266138910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd1980f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fd18c009fb0 tx=0x7fd18c009b30).stop 2023-12-08T10:15:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1266138910 shutdown_connections 2023-12-08T10:15:10.260 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 --2- 172.21.15.170:0/1266138910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd1980f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1266138910 >> 172.21.15.170:0/1266138910 conn(0x7fd1980f0b70 msgr2=0x7fd1980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1266138910 shutdown_connections 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1266138910 wait complete. 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 Processor -- start 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 -- start start 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd198106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.257+0000 7fd19cdd6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd198106c10 con 0x7fd1980f5410 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd198106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd198106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:59278/0 (socket says 172.21.15.170:59278) 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19659c700 1 -- 172.21.15.170:0/1026851425 learned_addr learned my addr 172.21.15.170:0/1026851425 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19659c700 1 -- 172.21.15.170:0/1026851425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c005040 con 0x7fd1980f5410 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19659c700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd198106720 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fd18c014000 tx=0x7fd18c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd18c0095f0 con 0x7fd1980f5410 2023-12-08T10:15:10.261 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd198106e10 con 0x7fd1980f5410 2023-12-08T10:15:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd18c007e70 con 0x7fd1980f5410 2023-12-08T10:15:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd18c014ea0 con 0x7fd1980f5410 2023-12-08T10:15:10.262 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.258+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1981072a0 con 0x7fd1980f5410 2023-12-08T10:15:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.259+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd18c025070 con 0x7fd1980f5410 2023-12-08T10:15:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.259+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd198045ad0 con 0x7fd1980f5410 2023-12-08T10:15:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.259+0000 7fd186ffd700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd18002ff60 0x7fd180032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.259+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd18c040c80 con 0x7fd1980f5410 2023-12-08T10:15:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.259+0000 7fd195d9b700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd18002ff60 0x7fd180032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:10.263 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.260+0000 7fd195d9b700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd18002ff60 0x7fd180032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd1880096a0 tx=0x7fd188006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:10.266 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.264+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd18c025950 con 0x7fd1980f5410 2023-12-08T10:15:10.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:09 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:08.389833+0000 mgr.smithi023.tgiigg (mgr.14164) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:10.470 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.468+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd198108150 con 0x7fd1980f5410 2023-12-08T10:15:10.471 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.469+0000 7fd186ffd700 1 -- 172.21.15.170:0/1026851425 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd18c025950 con 0x7fd1980f5410 2023-12-08T10:15:10.471 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:10.471 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:10.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd18002ff60 msgr2=0x7fd180032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:10.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd18002ff60 0x7fd180032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd1880096a0 tx=0x7fd188006b40).stop 2023-12-08T10:15:10.472 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 msgr2=0x7fd198106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd198106720 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fd18c014000 tx=0x7fd18c00b040).stop 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 shutdown_connections 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd18002ff60 0x7fd180032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 --2- 172.21.15.170:0/1026851425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd1980f5410 0x7fd198106720 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 >> 172.21.15.170:0/1026851425 conn(0x7fd1980f0b70 msgr2=0x7fd19805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 shutdown_connections 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:10.470+0000 7fd19cdd6700 1 -- 172.21.15.170:0/1026851425 wait complete. 2023-12-08T10:15:10.473 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:11.434 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:10 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:10.470334+0000 mon.smithi023 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.170:0/1026851425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:11.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:11.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:12.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:11 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:10.390103+0000 mgr.smithi023.tgiigg (mgr.14164) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:13.024 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 -- 172.21.15.170:0/749302282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 msgr2=0x7f1eb40f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:13.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 --2- 172.21.15.170:0/749302282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb40f6ea0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f1ea8009fb0 tx=0x7f1ea8009b30).stop 2023-12-08T10:15:13.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 -- 172.21.15.170:0/749302282 shutdown_connections 2023-12-08T10:15:13.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 --2- 172.21.15.170:0/749302282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb40f6ea0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:13.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 -- 172.21.15.170:0/749302282 >> 172.21.15.170:0/749302282 conn(0x7f1eb40f02e0 msgr2=0x7f1eb40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:13.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 -- 172.21.15.170:0/749302282 shutdown_connections 2023-12-08T10:15:13.025 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.022+0000 7f1eb9b56700 1 -- 172.21.15.170:0/749302282 wait complete. 2023-12-08T10:15:13.026 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb9b56700 1 Processor -- start 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb9b56700 1 -- start start 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb9b56700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb9b56700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4106bf0 con 0x7f1eb40f6a80 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:57142/0 (socket says 172.21.15.170:57142) 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.023+0000 7f1eb37fe700 1 -- 172.21.15.170:0/3556298261 learned_addr learned my addr 172.21.15.170:0/3556298261 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1eb37fe700 1 -- 172.21.15.170:0/3556298261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8005040 con 0x7f1eb40f6a80 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1eb37fe700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb4106700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f1ea8013510 tx=0x7f1ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea8009320 con 0x7f1eb40f6a80 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb4106df0 con 0x7f1eb40f6a80 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ea8013f00 con 0x7f1eb40f6a80 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ea8014dc0 con 0x7f1eb40f6a80 2023-12-08T10:15:13.027 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.024+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb4107280 con 0x7f1eb40f6a80 2023-12-08T10:15:13.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.025+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ea80095f0 con 0x7f1eb40f6a80 2023-12-08T10:15:13.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.025+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb4045ad0 con 0x7f1eb40f6a80 2023-12-08T10:15:13.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.025+0000 7f1e9bfff700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1e9c02ff60 0x7f1e9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:13.028 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.025+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ea8041000 con 0x7f1eb40f6a80 2023-12-08T10:15:13.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.025+0000 7f1eb2ffd700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1e9c02ff60 0x7f1e9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:13.029 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.026+0000 7f1eb2ffd700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1e9c02ff60 0x7f1e9c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1ea40096a0 tx=0x7f1ea4006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:13.032 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.030+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ea806d2b0 con 0x7f1eb40f6a80 2023-12-08T10:15:13.237 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.235+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1eb4108130 con 0x7f1eb40f6a80 2023-12-08T10:15:13.238 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.236+0000 7f1e9bfff700 1 -- 172.21.15.170:0/3556298261 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ea8010230 con 0x7f1eb40f6a80 2023-12-08T10:15:13.238 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:13.238 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.237+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1e9c02ff60 msgr2=0x7f1e9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.237+0000 7f1eb9b56700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1e9c02ff60 0x7f1e9c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1ea40096a0 tx=0x7f1ea4006b40).stop 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.237+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 msgr2=0x7f1eb4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.237+0000 7f1eb9b56700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb4106700 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f1ea8013510 tx=0x7f1ea800b040).stop 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.238+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 shutdown_connections 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.238+0000 7f1eb9b56700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7f1e9c02ff60 0x7f1e9c032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:13.240 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.238+0000 7f1eb9b56700 1 --2- 172.21.15.170:0/3556298261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eb40f6a80 0x7f1eb4106700 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:13.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.238+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 >> 172.21.15.170:0/3556298261 conn(0x7f1eb40f02e0 msgr2=0x7f1eb40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:13.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.238+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 shutdown_connections 2023-12-08T10:15:13.241 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:13.238+0000 7f1eb9b56700 1 -- 172.21.15.170:0/3556298261 wait complete. 2023-12-08T10:15:13.241 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:14.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:12.390388+0000 mgr.smithi023.tgiigg (mgr.14164) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:14.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:14 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:13.237581+0000 mon.smithi023 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.170:0/3556298261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:14.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:14.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:15.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 -- 172.21.15.170:0/2394447332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 msgr2=0x7fdd400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:15.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 --2- 172.21.15.170:0/2394447332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd400f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fdd34009fb0 tx=0x7fdd34009b30).stop 2023-12-08T10:15:15.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 -- 172.21.15.170:0/2394447332 shutdown_connections 2023-12-08T10:15:15.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 --2- 172.21.15.170:0/2394447332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd400f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:15.794 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 -- 172.21.15.170:0/2394447332 >> 172.21.15.170:0/2394447332 conn(0x7fdd400f0b70 msgr2=0x7fdd400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 -- 172.21.15.170:0/2394447332 shutdown_connections 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.790+0000 7fdd452d2700 1 -- 172.21.15.170:0/2394447332 wait complete. 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd452d2700 1 Processor -- start 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd452d2700 1 -- start start 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd452d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd40109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd452d2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40060240 con 0x7fdd400f5410 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd3effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd40109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd3effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd40109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:57172/0 (socket says 172.21.15.170:57172) 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd3effd700 1 -- 172.21.15.170:0/3094132806 learned_addr learned my addr 172.21.15.170:0/3094132806 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd3effd700 1 -- 172.21.15.170:0/3094132806 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34005040 con 0x7fdd400f5410 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.791+0000 7fdd3effd700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd40109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fdd34013510 tx=0x7fdd3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.792+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd34009320 con 0x7fdd400f5410 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.792+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd40060440 con 0x7fdd400f5410 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.792+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd34013f00 con 0x7fdd400f5410 2023-12-08T10:15:15.795 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.792+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd34014d90 con 0x7fdd400f5410 2023-12-08T10:15:15.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.792+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd40109ae0 con 0x7fdd400f5410 2023-12-08T10:15:15.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.793+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd340095f0 con 0x7fdd400f5410 2023-12-08T10:15:15.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.793+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd40045ad0 con 0x7fdd400f5410 2023-12-08T10:15:15.796 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.793+0000 7fdd277fe700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd2802ff60 0x7fdd28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:15.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.793+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd34040fe0 con 0x7fdd400f5410 2023-12-08T10:15:15.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.793+0000 7fdd3e7fc700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd2802ff60 0x7fdd28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:15.797 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.793+0000 7fdd3e7fc700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd2802ff60 0x7fdd28032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fdd300096a0 tx=0x7fdd30006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:15.800 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:15.798+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd3401d7f0 con 0x7fdd400f5410 2023-12-08T10:15:16.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.008+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd400fae60 con 0x7fdd400f5410 2023-12-08T10:15:16.010 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.008+0000 7fdd277fe700 1 -- 172.21.15.170:0/3094132806 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd34010230 con 0x7fdd400f5410 2023-12-08T10:15:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:16.011 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd2802ff60 msgr2=0x7fdd28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd2802ff60 0x7fdd28032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fdd300096a0 tx=0x7fdd30006b40).stop 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 msgr2=0x7fdd40109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd40109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fdd34013510 tx=0x7fdd3400b040).stop 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 shutdown_connections 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fdd2802ff60 0x7fdd28032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:16.012 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 --2- 172.21.15.170:0/3094132806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd400f5410 0x7fdd40109430 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:16.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 >> 172.21.15.170:0/3094132806 conn(0x7fdd400f0b70 msgr2=0x7fdd400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:16.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 shutdown_connections 2023-12-08T10:15:16.013 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:16.010+0000 7fdd452d2700 1 -- 172.21.15.170:0/3094132806 wait complete. 2023-12-08T10:15:16.013 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:16.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:16 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:14.390737+0000 mgr.smithi023.tgiigg (mgr.14164) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:17.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-08T10:15:17.404 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 -- ceph mon dump -f json 2023-12-08T10:15:17.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:17 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:16.010073+0000 mon.smithi023 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.170:0/3094132806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:18.432 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:18 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: cluster 2023-12-08T10:15:16.390980+0000 mgr.smithi023.tgiigg (mgr.14164) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-08T10:15:18.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 -- 172.21.15.170:0/3064886259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 msgr2=0x7fd6080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:18.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 --2- 172.21.15.170:0/3064886259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd6080f6930 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fd5f8009fb0 tx=0x7fd5f8009b30).stop 2023-12-08T10:15:18.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 -- 172.21.15.170:0/3064886259 shutdown_connections 2023-12-08T10:15:18.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 --2- 172.21.15.170:0/3064886259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd6080f6930 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:18.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 -- 172.21.15.170:0/3064886259 >> 172.21.15.170:0/3064886259 conn(0x7fd6080f1c70 msgr2=0x7fd6080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:18.538 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 -- 172.21.15.170:0/3064886259 shutdown_connections 2023-12-08T10:15:18.539 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 -- 172.21.15.170:0/3064886259 wait complete. 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 Processor -- start 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.536+0000 7fd60d77b700 1 -- start start 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd60d77b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd608106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd60d77b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd608106c20 con 0x7fd6080f6510 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd606ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd608106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd606ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd608106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.170:57190/0 (socket says 172.21.15.170:57190) 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd606ffd700 1 -- 172.21.15.170:0/1908565836 learned_addr learned my addr 172.21.15.170:0/1908565836 (peer_addr_for_me v2:172.21.15.170:0/0) 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd606ffd700 1 -- 172.21.15.170:0/1908565836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f8005040 con 0x7fd6080f6510 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd606ffd700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd608106730 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fd5f8013510 tx=0x7fd5f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:18.540 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5f8009320 con 0x7fd6080f6510 2023-12-08T10:15:18.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd608106e20 con 0x7fd6080f6510 2023-12-08T10:15:18.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5f8013f00 con 0x7fd6080f6510 2023-12-08T10:15:18.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5f8014d90 con 0x7fd6080f6510 2023-12-08T10:15:18.541 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.537+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6081072b0 con 0x7fd6080f6510 2023-12-08T10:15:18.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.538+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608045ad0 con 0x7fd6080f6510 2023-12-08T10:15:18.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.538+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5f80095f0 con 0x7fd6080f6510 2023-12-08T10:15:18.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.538+0000 7fd5ef7fe700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd5f002ff60 0x7fd5f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-08T10:15:18.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.538+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5f8040fc0 con 0x7fd6080f6510 2023-12-08T10:15:18.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.539+0000 7fd6067fc700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd5f002ff60 0x7fd5f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-08T10:15:18.542 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.539+0000 7fd6067fc700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd5f002ff60 0x7fd5f0032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd5fc0096a0 tx=0x7fd5fc006b40).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-08T10:15:18.545 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.543+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5f8014420 con 0x7fd6080f6510 2023-12-08T10:15:18.751 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.749+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6080fbf60 con 0x7fd6080f6510 2023-12-08T10:15:18.752 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.750+0000 7fd5ef7fe700 1 -- 172.21.15.170:0/1908565836 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5f8010230 con 0x7fd6080f6510 2023-12-08T10:15:18.752 INFO:teuthology.orchestra.run.smithi170.stdout: 2023-12-08T10:15:18.752 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"40b80ed0-95b1-11ee-95a3-87774f69a715","modified":"2023-12-08T10:05:27.678553Z","created":"2023-12-08T10:05:27.678553Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi023","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd5f002ff60 msgr2=0x7fd5f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd5f002ff60 0x7fd5f0032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd5fc0096a0 tx=0x7fd5fc006b40).stop 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 msgr2=0x7fd608106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd608106730 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fd5f8013510 tx=0x7fd5f800b040).stop 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 shutdown_connections 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:6800/2079584419,v1:172.21.15.23:6801/2079584419] conn(0x7fd5f002ff60 0x7fd5f0032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 --2- 172.21.15.170:0/1908565836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6080f6510 0x7fd608106730 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 >> 172.21.15.170:0/1908565836 conn(0x7fd6080f1c70 msgr2=0x7fd6080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 shutdown_connections 2023-12-08T10:15:18.754 INFO:teuthology.orchestra.run.smithi170.stderr:2023-12-08T10:15:18.751+0000 7fd60d77b700 1 -- 172.21.15.170:0/1908565836 wait complete. 2023-12-08T10:15:18.755 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2023-12-08T10:15:19.125 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-12-08T10:15:19.128 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-08T10:15:19.128 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-08T10:15:19.163 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-08T10:15:19.187 INFO:tasks.cephadm:Stopping all daemons... 2023-12-08T10:15:19.188 INFO:tasks.cephadm.mon.smithi023:Stopping mon.smithi023... 2023-12-08T10:15:19.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023 2023-12-08T10:15:19.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: audit 2023-12-08T10:15:18.751440+0000 mon.smithi023 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.170:0/1908565836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-08T10:15:19.303 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:19 smithi023 systemd[1]: Stopping Ceph mon.smithi023 for 40b80ed0-95b1-11ee-95a3-87774f69a715... 2023-12-08T10:15:19.304 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:15:19.300+0000 7f191e8a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi023 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-08T10:15:19.611 INFO:journalctl@ceph.mon.smithi023.smithi023.stdout:Dec 08 10:15:19 smithi023 ceph-40b80ed0-95b1-11ee-95a3-87774f69a715-mon.smithi023[111740]: debug 2023-12-08T10:15:19.300+0000 7f191e8a5700 -1 mon.smithi023@0(leader) e1 *** Got Signal Terminated *** 2023-12-08T10:15:19.777 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi023.service' 2023-12-08T10:15:19.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-08T10:15:19.837 INFO:tasks.cephadm.mon.smithi023:Stopped mon.smithi023 2023-12-08T10:15:19.837 INFO:tasks.cephadm.mon.smithi170:Stopping mon.smithi170... 2023-12-08T10:15:19.837 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170 2023-12-08T10:15:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-08T10:15:19.878 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170.service: Unit ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170.service not loaded. 2023-12-08T10:15:19.879 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi170" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 5: 'sudo systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170' 2023-12-08T10:15:19.880 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-08T10:15:19.884 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/remote/ubuntu@smithi023.front.sepia.ceph.com/crash 2023-12-08T10:15:19.886 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/crash -- . 2023-12-08T10:15:19.943 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/remote/ubuntu@smithi170.front.sepia.ceph.com/crash 2023-12-08T10:15:19.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/crash -- . 2023-12-08T10:15:19.994 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/crash: Cannot open: No such file or directory 2023-12-08T10:15:19.995 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2023-12-08T10:15:19.995 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-08T10:15:19.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-08T10:15:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:grep: /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph.log: No such file or directory 2023-12-08T10:15:20.033 INFO:tasks.cephadm:Compressing logs... 2023-12-08T10:15:20.033 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-08T10:15:20.074 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-08T10:15:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-08T10:15:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2023-12-08T10:15:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-volume.log 2023-12-08T10:15:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mgr.smithi023.tgiigg.log 2023-12-08T10:15:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mon.smithi023.log 2023-12-08T10:15:20.103 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-volume.log: /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mgr.smithi023.tgiigg.log: /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mon.smithi023.log: 90.3% -- replaced with /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-volume.log.gz 2023-12-08T10:15:20.104 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-08T10:15:20.104 INFO:teuthology.orchestra.run.smithi170.stderr:: No such file or directory 2023-12-08T10:15:20.106 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-08T10:15:20.107 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-12-08T10:15:20.107 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m0.014s 2023-12-08T10:15:20.108 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m0.008s 2023-12-08T10:15:20.108 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.014s 2023-12-08T10:15:20.108 INFO:teuthology.orchestra.run.smithi023.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-08T10:15:20.180 INFO:teuthology.orchestra.run.smithi023.stderr:gzip: /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mgr.smithi023.tgiigg.log: file size changed while zipping 2023-12-08T10:15:20.181 INFO:teuthology.orchestra.run.smithi023.stderr: 92.7% -- replaced with /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mgr.smithi023.tgiigg.log.gz 2023-12-08T10:15:20.251 INFO:teuthology.orchestra.run.smithi023.stderr: 90.5% -- replaced with /var/log/ceph/40b80ed0-95b1-11ee-95a3-87774f69a715/ceph-mon.smithi023.log.gz 2023-12-08T10:15:20.253 INFO:teuthology.orchestra.run.smithi023.stderr: 2023-12-08T10:15:20.253 INFO:teuthology.orchestra.run.smithi023.stderr:real 0m0.163s 2023-12-08T10:15:20.253 INFO:teuthology.orchestra.run.smithi023.stderr:user 0m0.239s 2023-12-08T10:15:20.253 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m0.018s 2023-12-08T10:15:20.253 INFO:tasks.cephadm:Archiving logs... 2023-12-08T10:15:20.253 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/remote/ubuntu@smithi023.front.sepia.ceph.com/log 2023-12-08T10:15:20.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-08T10:15:20.370 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/remote/ubuntu@smithi170.front.sepia.ceph.com/log 2023-12-08T10:15:20.371 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-08T10:15:20.401 INFO:tasks.cephadm:Removing cluster... 2023-12-08T10:15:20.401 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 --force 2023-12-08T10:16:48.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 40b80ed0-95b1-11ee-95a3-87774f69a715 --force 2023-12-08T10:16:48.343 INFO:tasks.cephadm:Removing cephadm ... 2023-12-08T10:16:48.343 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-08T10:16:48.363 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-08T10:16:48.381 INFO:tasks.cephadm:Teardown complete 2023-12-08T10:16:48.381 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 5: 'sudo systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170' 2023-12-08T10:16:48.467 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e0a794ef636545f3b3682c259792cb99 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi170 with status 5: 'sudo systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170' 2023-12-08T10:16:48.472 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-08T10:16:48.483 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-08T10:16:48.491 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-08T10:16:48.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-08T10:16:48.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-08T10:16:48.598 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-08T10:16:48.607 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-08T10:16:48.607 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T10:16:48.609 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-08T10:16:48.625 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2023-12-08T10:16:48.626 INFO:teuthology.orchestra.run.smithi170.stderr:bash: ntpq: command not found 2023-12-08T10:16:48.631 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-08T10:16:48.631 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2023-12-08T10:16:48.631 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +983us[ +961us] +/- 83ms 2023-12-08T10:16:48.631 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -216us[ -238us] +/- 48ms 2023-12-08T10:16:48.631 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -703us[ -725us] +/- 94ms 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +989us[ +987us] +/- 83ms 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -217us[ -219us] +/- 49ms 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -712us[ -712us] +/- 95ms 2023-12-08T10:16:48.632 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-08T10:16:48.633 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-08T10:16:48.642 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-08T10:16:48.642 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-08T10:16:48.650 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-08T10:16:48.762 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-08T10:16:48.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-08T10:16:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T10:16:48.884 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-08T10:16:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T10:16:48.918 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-08T10:16:48.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-08T10:16:48.977 INFO:teuthology.task.internal:Duration was 1393.075162 seconds 2023-12-08T10:16:48.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-08T10:16:48.987 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-08T10:16:48.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-08T10:16:48.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-08T10:16:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-08T10:16:49.035 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-08T10:16:49.194 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-08T10:16:49.194 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2023-12-08T10:16:49.194 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-08T10:16:49.217 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2023-12-08T10:16:49.218 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-08T10:16:49.240 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-08T10:16:49.240 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-08T10:16:49.260 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-08T10:16:49.327 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-08T10:16:49.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-08T10:16:49.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-08T10:16:49.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-08T10:16:49.464 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-08T10:16:49.464 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-08T10:16:49.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-08T10:16:49.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-08T10:16:49.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-08T10:16:49.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-08T10:16:49.560 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2023-12-08T10:16:49.586 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2023-12-08T10:16:49.627 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-08T10:16:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T10:16:49.655 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-08T10:16:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-08T10:16:49.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-08T10:16:49.696 INFO:teuthology.task.internal:Transferring archived files... 2023-12-08T10:16:49.696 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/remote/smithi023 2023-12-08T10:16:49.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-08T10:16:49.740 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419/remote/smithi170 2023-12-08T10:16:49.741 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-08T10:16:49.780 INFO:teuthology.task.internal:Removing archive directory... 2023-12-08T10:16:49.780 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-08T10:16:49.782 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-08T10:16:49.833 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-08T10:16:49.844 INFO:teuthology.task.internal:Not uploading archives. 2023-12-08T10:16:49.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-08T10:16:49.855 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-08T10:16:49.855 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-08T10:16:49.857 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-08T10:16:49.873 INFO:teuthology.orchestra.run.smithi023.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 8 10:16 /home/ubuntu/cephtest 2023-12-08T10:16:49.890 INFO:teuthology.orchestra.run.smithi170.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 8 10:16 /home/ubuntu/cephtest 2023-12-08T10:16:49.892 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-08T10:16:49.905 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-08T10:16:49.928 INFO:teuthology.nuke:Checking targets against current locks 2023-12-08T10:16:49.955 DEBUG:teuthology.nuke:shortname: smithi023 2023-12-08T10:16:49.955 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-08T10:16:49.979 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-08 09:43:33.060496', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD2mTZcolWLO2JNlYBeDpOo4vqvQ8kcgIwCIB0zlJ5HgDhlAsVAgKJaqSAuAdFR5tSdOxXPulHbdV4ejODiGtvg='} 2023-12-08T10:16:49.983 DEBUG:teuthology.nuke:shortname: smithi170 2023-12-08T10:16:49.983 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-08T10:16:50.005 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-08_04:25:47-rados-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7483419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-08 09:43:33.061947', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDp4zu2Wjlz0Wu3gTPx5rr7Z/Nhh+9F19811qM/0lEvQ51PG9q+ys4Im0cOWf2L6KOMXlfZpeXo3dFSN1s2kBGo='} 2023-12-08T10:16:50.040 INFO:teuthology.orchestra.console.smithi023:Power off 2023-12-08T10:16:50.040 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-08T10:16:50.064 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2023-12-08T10:16:50.064 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T10:16:50.081 INFO:teuthology.orchestra.console.smithi170:Power off 2023-12-08T10:16:50.082 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-08T10:16:50.090 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2023-12-08T10:16:50.107 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2023-12-08T10:16:50.107 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T10:16:50.130 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-08T10:16:54.091 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T10:16:54.131 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T10:16:54.202 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2023-12-08T10:16:54.241 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2023-12-08T10:16:58.206 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T10:16:58.242 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-08T10:16:58.317 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2023-12-08T10:16:58.354 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2023-12-08T10:16:58.418 INFO:teuthology.orchestra.console.smithi023:Power off completed 2023-12-08T10:16:58.455 INFO:teuthology.orchestra.console.smithi170:Power off completed 2023-12-08T10:16:58.560 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1393.0751616954803 failure_reason: 'Command failed on smithi170 with status 5: ''sudo systemctl stop ceph-40b80ed0-95b1-11ee-95a3-87774f69a715@mon.smithi170''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e0a794ef636545f3b3682c259792cb99 status: fail success: false 2023-12-08T10:16:58.561 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-08T10:16:58.649 INFO:teuthology.run:FAIL