2024-02-10T09:03:03.463 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T09:03:03.463 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366 -- /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/orig.config.yaml 2024-02-10T09:03:03.512 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T09:03:03.606 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7555366' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2minzNyxFmKUjaJtZ6ACDSKDWopfG+HFFA78GefUSuRxQm10cELFiLRhYunEL8YitTj5Pngtsvn3MihqKMFDg= smithi158.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCT6NtxExKSHXAwbMTl1REWH7NxSpl6ZC+wdPrXXQDRwf6PdwEjwUWWXCVdkiLKwQuJbVmw52XGnMSi8hkRNxqs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_21:34:44 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-10T09:03:03.606 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-10T09:03:03.607 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-10T09:03:03.607 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T09:03:03.609 INFO:teuthology.task.internal:Checking packages... 2024-02-10T09:03:03.630 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-10T09:03:03.630 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T09:03:03.630 INFO:teuthology.packaging:ref: None 2024-02-10T09:03:03.630 INFO:teuthology.packaging:tag: None 2024-02-10T09:03:03.631 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T09:03:03.631 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:03:03.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T09:03:03.805 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-10T09:03:03.806 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T09:03:03.812 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T09:03:03.812 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T09:03:03.818 INFO:teuthology.task.internal:Saving configuration 2024-02-10T09:03:03.836 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T09:03:03.841 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:03:03.867 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:45.924660', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2minzNyxFmKUjaJtZ6ACDSKDWopfG+HFFA78GefUSuRxQm10cELFiLRhYunEL8YitTj5Pngtsvn3MihqKMFDg='} 2024-02-10T09:03:03.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:45.922682', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCT6NtxExKSHXAwbMTl1REWH7NxSpl6ZC+wdPrXXQDRwf6PdwEjwUWWXCVdkiLKwQuJbVmw52XGnMSi8hkRNxqs='} 2024-02-10T09:03:03.896 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T09:03:03.903 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-10T09:03:03.903 INFO:teuthology.task.internal:roles: ubuntu@smithi158.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-10T09:03:03.903 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T09:03:03.965 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4df8911f70>, signals=[15]) 2024-02-10T09:03:03.965 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T09:03:03.971 INFO:teuthology.task.internal:Opening connections... 2024-02-10T09:03:03.972 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-02-10T09:03:03.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:03:04.050 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi158.front.sepia.ceph.com 2024-02-10T09:03:04.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:03:04.122 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T09:03:04.130 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-02-10T09:03:04.153 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-02-10T09:03:04.153 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-02-10T09:03:04.214 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="8" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="8" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el8" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T09:03:04.215 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T09:03:04.216 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-02-10T09:03:04.235 DEBUG:teuthology.orchestra.run.smithi158:> uname -m 2024-02-10T09:03:04.257 INFO:teuthology.orchestra.run.smithi158.stdout:x86_64 2024-02-10T09:03:04.258 DEBUG:teuthology.orchestra.run.smithi158:> cat /etc/os-release 2024-02-10T09:03:04.318 INFO:teuthology.orchestra.run.smithi158.stdout:NAME="CentOS Stream" 2024-02-10T09:03:04.318 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION="8" 2024-02-10T09:03:04.318 INFO:teuthology.orchestra.run.smithi158.stdout:ID="centos" 2024-02-10T09:03:04.318 INFO:teuthology.orchestra.run.smithi158.stdout:ID_LIKE="rhel fedora" 2024-02-10T09:03:04.318 INFO:teuthology.orchestra.run.smithi158.stdout:VERSION_ID="8" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:PLATFORM_ID="platform:el8" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:ANSI_COLOR="0;31" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:HOME_URL="https://centos.org/" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T09:03:04.319 INFO:teuthology.orchestra.run.smithi158.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T09:03:04.320 INFO:teuthology.lock.ops:Updating smithi158.front.sepia.ceph.com on lock server 2024-02-10T09:03:04.343 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T09:03:04.353 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T09:03:04.358 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T09:03:04.358 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-02-10T09:03:04.360 DEBUG:teuthology.orchestra.run.smithi158:> test '!' -e /home/ubuntu/cephtest 2024-02-10T09:03:04.380 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T09:03:04.385 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T09:03:04.385 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-02-10T09:03:04.422 DEBUG:teuthology.orchestra.run.smithi158:> test -z $(ls -A /var/lib/ceph) 2024-02-10T09:03:04.454 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T09:03:04.512 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T09:03:04.527 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T09:03:04.527 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T09:03:04.528 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T09:03:04.528 DEBUG:teuthology.orchestra.run.smithi158:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T09:03:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:03:04.548 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-02-10T09:03:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:03:04.549 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-02-10T09:03:04.609 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-489.el8.x86_64 2024-02-10T09:03:04.610 INFO:teuthology.task.kernel:Running kernel on smithi158: 4.18.0-489.el8.x86_64 2024-02-10T09:03:04.610 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum install -y kernel 2024-02-10T09:03:04.611 INFO:teuthology.orchestra.run.smithi083.stdout:4.18.0-489.el8.x86_64 2024-02-10T09:03:04.611 INFO:teuthology.task.kernel:Running kernel on smithi083: 4.18.0-489.el8.x86_64 2024-02-10T09:03:04.611 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-02-10T09:03:12.411 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - AppStream 5.2 MB/s | 27 MB 00:05 2024-02-10T09:03:18.386 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - BaseOS 8.4 MB/s | 10 MB 00:01 2024-02-10T09:03:20.170 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras 95 kB/s | 18 kB 00:00 2024-02-10T09:03:20.656 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-02-10T09:03:20.974 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-10T09:03:22.086 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-02-10T09:03:22.767 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - AppStream 1.7 MB/s | 27 MB 00:15 2024-02-10T09:03:25.800 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 552 kB/s | 24 kB 00:00 2024-02-10T09:03:28.722 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T09:03:28.723 INFO:teuthology.orchestra.run.smithi158.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T09:03:28.754 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - BaseOS 8.4 MB/s | 10 MB 00:01 2024-02-10T09:03:28.839 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:03:28.839 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:03:28.839 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout:Install 3 Packages 2024-02-10T09:03:28.840 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:03:28.841 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 90 M 2024-02-10T09:03:28.841 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 96 M 2024-02-10T09:03:28.841 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:03:30.019 INFO:teuthology.orchestra.run.smithi158.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 9.0 MB/s | 10 MB 00:01 2024-02-10T09:03:30.720 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-02-10T09:03:30.977 INFO:teuthology.orchestra.run.smithi158.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2024-02-10T09:03:31.188 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-10T09:03:31.465 INFO:teuthology.orchestra.run.smithi083.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-02-10T09:03:31.769 INFO:teuthology.orchestra.run.smithi158.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 15 MB/s | 44 MB 00:02 2024-02-10T09:03:31.769 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:03:31.769 INFO:teuthology.orchestra.run.smithi158.stdout:Total 31 MB/s | 90 MB 00:02 2024-02-10T09:03:32.448 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:03:32.871 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:03:32.872 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:03:34.102 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:03:34.128 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:03:36.954 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:03:37.467 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:03:41.076 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:03:41.517 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:03:41.701 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 1.6 MB/s | 16 MB 00:10 2024-02-10T09:03:44.961 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:03:45.258 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:03:45.407 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 510 kB/s | 24 kB 00:00 2024-02-10T09:03:48.287 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T09:03:48.287 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T09:03:48.411 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:03:48.412 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:03:48.412 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:03:48.412 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Install 3 Packages 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 90 M 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 96 M 2024-02-10T09:03:48.413 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:03:49.525 INFO:teuthology.orchestra.run.smithi083.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 9.9 MB/s | 10 MB 00:01 2024-02-10T09:03:50.492 INFO:teuthology.orchestra.run.smithi083.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-02-10T09:03:51.117 INFO:teuthology.orchestra.run.smithi083.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 17 MB/s | 44 MB 00:02 2024-02-10T09:03:51.118 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:03:51.118 INFO:teuthology.orchestra.run.smithi083.stdout:Total 33 MB/s | 90 MB 00:02 2024-02-10T09:03:51.801 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:03:52.209 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:03:52.209 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:03:53.455 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:03:53.477 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:03:56.262 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:03:56.814 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:00.365 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:00.755 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:04.199 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:04.546 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:07.213 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:07.602 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:07.603 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:04:08.154 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:04:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T09:04:08.154 INFO:teuthology.orchestra.run.smithi158.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T09:04:08.155 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:04:08.155 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:04:08.801 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:04:09.020 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:04:09.020 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:04:09.020 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T09:04:09.021 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T09:04:09.021 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T09:04:09.021 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-10T09:04:09.021 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi158.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T09:04:09.021 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:04:09.232 INFO:teuthology.orchestra.run.smithi158.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:04:09.232 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:04:09.232 DEBUG:teuthology.orchestra.run.smithi158:> sudo rpm -qi grub2-tools 2024-02-10T09:04:09.266 INFO:teuthology.orchestra.run.smithi158.stdout:Name : grub2-tools 2024-02-10T09:04:09.266 INFO:teuthology.orchestra.run.smithi158.stdout:Epoch : 1 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Version : 2.02 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Release : 129.el8 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Architecture: x86_64 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Group : System Environment/Base 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Size : 9265586 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:License : GPLv3+ 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Relocations : (not relocatable) 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Packager : CentOS BuildSystem 2024-02-10T09:04:09.267 INFO:teuthology.orchestra.run.smithi158.stdout:Vendor : CentOS 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:Summary : Support tools for GRUB. 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:Description : 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:hardware devices. 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:04:09.268 INFO:teuthology.orchestra.run.smithi158.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T09:04:09.269 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T09:04:09.269 INFO:teuthology.task.kernel:Updating grub on smithi158 to boot 4.18.0-536.el8.x86_64 2024-02-10T09:04:09.269 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T09:04:10.750 INFO:teuthology.orchestra.run.smithi158.stderr:Generating grub configuration file ... 2024-02-10T09:04:12.739 INFO:teuthology.orchestra.run.smithi158.stderr:done 2024-02-10T09:04:12.741 DEBUG:teuthology.orchestra.run.smithi158:> mktemp 2024-02-10T09:04:12.763 INFO:teuthology.orchestra.run.smithi158.stdout:/tmp/tmp.iIIlp96lhF 2024-02-10T09:04:12.763 DEBUG:teuthology.orchestra.run.smithi158:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iIIlp96lhF 2024-02-10T09:04:12.838 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 0666 /tmp/tmp.iIIlp96lhF 2024-02-10T09:04:12.974 DEBUG:teuthology.orchestra.remote:smithi158:/tmp/tmp.iIIlp96lhF is 6KB 2024-02-10T09:04:13.021 DEBUG:teuthology.orchestra.run.smithi158:> rm -fr /tmp/tmp.iIIlp96lhF 2024-02-10T09:04:13.039 DEBUG:teuthology.orchestra.run.smithi158:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T09:04:13.115 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T09:04:13.115 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T09:04:13.115 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T09:04:13.115 INFO:teuthology.orchestra.run.smithi158.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T09:04:13.116 DEBUG:teuthology.orchestra.run.smithi158:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T09:04:13.298 DEBUG:teuthology.orchestra.run.smithi158:> sudo shutdown -r now 2024-02-10T09:04:26.340 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:26.362 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T09:04:26.711 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:04:27.263 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:04:27.414 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:04:27.634 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:04:27.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:04:27.634 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T09:04:27.635 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T09:04:27.635 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T09:04:27.635 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-10T09:04:27.635 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T09:04:27.635 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T09:04:27.845 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T09:04:27.845 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T09:04:27.845 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.02 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 129.el8 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Group : System Environment/Base 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 9265586 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T09:04:27.879 INFO:teuthology.orchestra.run.smithi083.stdout:Relocations : (not relocatable) 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : CentOS BuildSystem 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:04:27.880 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T09:04:27.882 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T09:04:27.882 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 4.18.0-536.el8.x86_64 2024-02-10T09:04:27.882 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T09:04:29.722 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-02-10T09:04:31.633 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-02-10T09:04:31.635 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-02-10T09:04:31.674 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.8gFRtu55sD 2024-02-10T09:04:31.675 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8gFRtu55sD 2024-02-10T09:04:31.749 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.8gFRtu55sD 2024-02-10T09:04:31.906 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.8gFRtu55sD is 6KB 2024-02-10T09:04:31.954 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.8gFRtu55sD 2024-02-10T09:04:31.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T09:04:32.044 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T09:04:32.044 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T09:04:32.044 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T09:04:32.045 INFO:teuthology.orchestra.run.smithi083.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T09:04:32.046 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T09:04:32.208 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-02-10T09:04:43.312 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-10T09:04:43.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:04:43.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:01.756 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-10T09:05:02.210 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-10T09:05:02.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:05:02.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:10.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:05:10.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:13.824 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-10T09:05:20.700 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-02-10T09:05:25.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:05:25.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:28.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-10T09:05:29.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:05:29.703 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:32.764 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-02-10T09:05:43.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:05:43.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:44.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:05:44.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:05:47.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-02-10T09:06:02.432 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.158 2024-02-10T09:06:02.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:06:02.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:05.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-02-10T09:06:20.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:06:20.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:20.801 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-02-10T09:06:21.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:06:21.069 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-536.el8.x86_64"... 2024-02-10T09:06:21.070 DEBUG:teuthology.orchestra.run.smithi158:> uname -r 2024-02-10T09:06:21.087 INFO:teuthology.orchestra.run.smithi158.stdout:4.18.0-536.el8.x86_64 2024-02-10T09:06:21.087 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T09:06:21.087 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T09:06:21.087 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T09:06:22.088 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-10T09:06:22.088 DEBUG:teuthology.orchestra.run.smithi158:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T09:06:22.875 INFO:teuthology.orchestra.run.smithi158.stdout:ttyS1 2024-02-10T09:06:22.906 DEBUG:teuthology.parallel:result is None 2024-02-10T09:06:23.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:06:23.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:06:24.192 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-02-10T09:06:24.455 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:06:24.456 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-536.el8.x86_64"... 2024-02-10T09:06:24.456 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-02-10T09:06:24.516 INFO:teuthology.orchestra.run.smithi083.stdout:4.18.0-536.el8.x86_64 2024-02-10T09:06:24.516 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T09:06:24.516 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T09:06:24.516 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T09:06:25.517 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-10T09:06:25.517 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T09:06:25.605 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-02-10T09:06:25.617 DEBUG:teuthology.parallel:result is None 2024-02-10T09:06:25.618 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T09:06:25.625 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T09:06:25.625 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T09:06:25.628 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T09:06:25.653 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T09:06:25.660 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T09:06:25.666 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T09:06:25.666 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T09:06:25.688 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T09:06:25.729 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T09:06:25.736 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T09:06:25.737 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T09:06:25.758 DEBUG:teuthology.orchestra.run.smithi158:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T09:06:25.791 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:25.802 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:25.809 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:25.820 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T09:06:25.822 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T09:06:25.829 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T09:06:25.829 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T09:06:25.846 DEBUG:teuthology.orchestra.run.smithi158:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T09:06:25.876 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T09:06:25.892 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T09:06:25.892 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T09:06:25.917 DEBUG:teuthology.orchestra.run.smithi158:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T09:06:25.936 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:26.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:26.072 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:26.158 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:26.199 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:06:26.199 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T09:06:26.266 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:26.315 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T09:06:26.368 DEBUG:teuthology.orchestra.run.smithi158:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:26.455 DEBUG:teuthology.orchestra.run.smithi158:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T09:06:26.500 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:06:26.500 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T09:06:26.568 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-02-10T09:06:26.570 DEBUG:teuthology.orchestra.run.smithi158:> sudo service rsyslog restart 2024-02-10T09:06:26.632 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:06:26.671 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:06:27.037 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T09:06:27.046 INFO:teuthology.task.internal:Starting timer... 2024-02-10T09:06:27.046 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T09:06:27.068 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T09:06:27.076 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T09:06:27.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-02-10T09:06:27.198 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-02-10T09:06:27.200 DEBUG:teuthology.orchestra.run.smithi158:> sudo service auditd rotate 2024-02-10T09:06:27.280 INFO:teuthology.orchestra.run.smithi158.stdout:Rotating logs: 2024-02-10T09:06:27.281 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T09:06:27.281 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-02-10T09:06:27.302 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-02-10T09:06:27.303 DEBUG:teuthology.orchestra.run.smithi158:> /usr/sbin/getenforce 2024-02-10T09:06:27.355 INFO:teuthology.orchestra.run.smithi158.stdout:Permissive 2024-02-10T09:06:27.355 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi158.front.sepia.ceph.com': 'permissive'} 2024-02-10T09:06:27.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:06:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:06:27.382 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:06:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:06:27.409 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T09:06:27.410 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-02-10T09:06:27.450 DEBUG:teuthology.orchestra.run.smithi158:> sudo /usr/sbin/setenforce permissive 2024-02-10T09:06:27.476 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T09:06:27.490 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T09:06:27.507 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T09:06:27.508 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com,smithi158.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T09:14:44.607 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi158.front.sepia.ceph.com')] 2024-02-10T09:14:44.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:14:44.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:14:44.686 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-02-10T09:14:44.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-02-10T09:14:44.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:14:44.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi158.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T09:14:44.828 DEBUG:teuthology.orchestra.run.smithi158:> true 2024-02-10T09:14:44.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi158.front.sepia.ceph.com' 2024-02-10T09:14:44.899 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T09:14:44.910 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T09:14:44.911 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T09:14:44.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:14:44.914 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T09:14:44.914 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:14:44.943 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T09:14:44.956 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T09:14:44.971 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T09:14:44.985 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T09:14:45.020 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-02-10T09:14:45.030 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-02-10T09:14:45.044 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T09:14:45.049 INFO:teuthology.orchestra.run.smithi158.stderr:sudo: ntpd: command not found 2024-02-10T09:14:45.057 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T09:14:45.061 INFO:teuthology.orchestra.run.smithi158.stdout:506 Cannot talk to daemon 2024-02-10T09:14:45.074 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T09:14:45.087 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T09:14:45.120 INFO:teuthology.orchestra.run.smithi083.stderr:bash: ntpq: command not found 2024-02-10T09:14:45.124 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:14:45.125 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-02-10T09:14:45.125 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.125 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.125 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.125 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.148 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2024-02-10T09:14:45.152 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:14:45.152 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-02-10T09:14:45.152 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.152 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.152 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.152 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:14:45.153 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T09:14:45.162 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T09:14:45.162 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi083.front.sepia.ceph.com 2024-02-10T09:14:45.162 DEBUG:teuthology.orchestra.run.smithi083:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T09:14:45.163 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi158.front.sepia.ceph.com 2024-02-10T09:14:45.163 DEBUG:teuthology.orchestra.run.smithi158:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T09:14:45.942 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:20 ago on Sat 10 Feb 2024 09:13:25 AM UTC. 2024-02-10T09:14:45.972 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 09:13:12 AM UTC. 2024-02-10T09:14:47.862 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:14:47.862 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout:Resetting modules: 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout: container-tools 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:14:47.863 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:14:47.885 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:14:47.886 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout:Resetting modules: 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:14:47.887 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:14:48.112 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:14:48.142 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:14:48.949 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:23 ago on Sat 10 Feb 2024 09:13:25 AM UTC. 2024-02-10T09:14:49.001 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 09:13:12 AM UTC. 2024-02-10T09:14:50.940 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout:Installing group/module packages: 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T09:14:50.976 INFO:teuthology.orchestra.run.smithi083.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T09:14:50.977 INFO:teuthology.orchestra.run.smithi083.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T09:14:50.978 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-02-10T09:14:50.978 INFO:teuthology.orchestra.run.smithi083.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-10T09:14:50.978 INFO:teuthology.orchestra.run.smithi083.stdout:Installing module profiles: 2024-02-10T09:14:50.978 INFO:teuthology.orchestra.run.smithi083.stdout: container-tools/common 2024-02-10T09:14:50.979 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:14:50.979 INFO:teuthology.orchestra.run.smithi083.stdout:Enabling module streams: 2024-02-10T09:14:50.979 INFO:teuthology.orchestra.run.smithi083.stdout: container-tools rhel8 2024-02-10T09:14:50.979 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:14:50.979 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:14:50.980 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:14:50.980 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-02-10T09:14:50.980 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:14:50.981 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 27 M 2024-02-10T09:14:50.981 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 91 M 2024-02-10T09:14:50.981 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:14:50.989 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:14:51.024 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout:Installing group/module packages: 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T09:14:51.025 INFO:teuthology.orchestra.run.smithi158.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout:Installing weak dependencies: 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout:Installing module profiles: 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools/common 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout:Enabling module streams: 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: container-tools rhel8 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:14:51.026 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:14:51.027 INFO:teuthology.orchestra.run.smithi158.stdout:Install 13 Packages 2024-02-10T09:14:51.027 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:14:51.027 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 27 M 2024-02-10T09:14:51.027 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 91 M 2024-02-10T09:14:51.027 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:14:51.340 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 2.3 MB/s | 759 kB 00:00 2024-02-10T09:14:51.399 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 650 kB/s | 38 kB 00:00 2024-02-10T09:14:51.404 INFO:teuthology.orchestra.run.smithi158.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 2.2 MB/s | 759 kB 00:00 2024-02-10T09:14:51.454 INFO:teuthology.orchestra.run.smithi158.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 2.6 MB/s | 1.0 MB 00:00 2024-02-10T09:14:51.457 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): aardvark-dns-1.9.0-1.module_el8+804+f13 2.3 MB/s | 1.0 MB 00:00 2024-02-10T09:14:51.480 INFO:teuthology.orchestra.run.smithi158.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 503 kB/s | 38 kB 00:00 2024-02-10T09:14:51.482 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 2.9 MB/s | 248 kB 00:00 2024-02-10T09:14:51.539 INFO:teuthology.orchestra.run.smithi158.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 2.9 MB/s | 248 kB 00:00 2024-02-10T09:14:51.599 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.5 MB/s | 179 kB 00:00 2024-02-10T09:14:51.639 INFO:teuthology.orchestra.run.smithi158.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 1.8 MB/s | 179 kB 00:00 2024-02-10T09:14:51.683 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 297 kB/s | 25 kB 00:00 2024-02-10T09:14:51.697 INFO:teuthology.orchestra.run.smithi158.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 425 kB/s | 25 kB 00:00 2024-02-10T09:14:51.758 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-02-10T09:14:51.772 INFO:teuthology.orchestra.run.smithi158.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-02-10T09:14:51.942 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): buildah-1.34.0-1.module_el8+804+f131391 10 MB/s | 9.7 MB 00:00 2024-02-10T09:14:52.031 INFO:teuthology.orchestra.run.smithi158.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 7.2 MB/s | 4.0 MB 00:00 2024-02-10T09:14:52.558 INFO:teuthology.orchestra.run.smithi158.stdout:(9/13): toolbox-0.0.99.4-8.module_el8+804+f1313 4.8 MB/s | 2.5 MB 00:00 2024-02-10T09:14:52.682 INFO:teuthology.orchestra.run.smithi158.stdout:(10/13): buildah-1.34.0-1.module_el8+804+f13139 6.0 MB/s | 9.7 MB 00:01 2024-02-10T09:14:52.684 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): netavark-1.9.0-1.module_el8+804+f131391 3.2 MB/s | 4.0 MB 00:01 2024-02-10T09:14:52.707 INFO:teuthology.orchestra.run.smithi158.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 353 kB/s | 53 kB 00:00 2024-02-10T09:14:52.749 INFO:teuthology.orchestra.run.smithi158.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 615 kB/s | 41 kB 00:00 2024-02-10T09:14:52.759 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 702 kB/s | 53 kB 00:00 2024-02-10T09:14:52.819 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 701 kB/s | 41 kB 00:00 2024-02-10T09:14:52.934 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): skopeo-1.14.0-2.module_el8+804+f131391 7.5 MB/s | 8.8 MB 00:01 2024-02-10T09:14:52.974 INFO:teuthology.orchestra.run.smithi158.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 7.3 MB/s | 8.8 MB 00:01 2024-02-10T09:14:52.974 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:14:52.975 INFO:teuthology.orchestra.run.smithi158.stdout:Total 14 MB/s | 27 MB 00:01 2024-02-10T09:14:53.244 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:14:53.265 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:14:53.265 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:14:53.485 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): toolbox-0.0.99.4-8.module_el8+804+f131 1.6 MB/s | 2.5 MB 00:01 2024-02-10T09:14:53.485 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:14:53.485 INFO:teuthology.orchestra.run.smithi083.stdout:Total 11 MB/s | 27 MB 00:02 2024-02-10T09:14:53.522 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:14:53.525 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:14:53.758 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:14:53.779 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:14:53.779 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:14:54.062 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:14:54.065 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:14:54.525 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:14:54.752 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:14:54.763 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-10T09:14:54.942 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-10T09:14:54.981 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-10T09:14:55.159 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-10T09:14:55.163 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-10T09:14:55.356 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-10T09:14:55.394 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T09:14:55.601 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T09:14:55.933 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-10T09:14:56.183 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-10T09:14:56.948 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T09:14:57.182 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T09:14:57.192 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-10T09:14:57.413 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-10T09:14:57.419 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-10T09:14:57.612 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-10T09:14:57.696 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-10T09:14:57.930 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-10T09:14:58.112 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-10T09:14:58.322 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-10T09:14:59.068 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T09:14:59.280 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-10T09:14:59.284 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T09:14:59.549 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T09:14:59.556 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-10T09:14:59.741 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T09:15:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T09:15:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-10T09:15:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-10T09:15:00.002 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T09:15:00.003 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-10T09:15:00.165 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T09:15:00.166 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-10T09:15:00.498 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-10T09:15:00.498 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:15:00.498 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:15:00.499 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:15:00.675 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-10T09:15:00.675 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:15:00.675 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:15:00.675 INFO:teuthology.orchestra.run.smithi083.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:15:00.676 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:15:00.784 DEBUG:teuthology.parallel:result is None 2024-02-10T09:15:00.984 DEBUG:teuthology.parallel:result is None 2024-02-10T09:15:00.984 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T09:15:00.995 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T09:15:00.996 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi083.front.sepia.ceph.com 2024-02-10T09:15:00.996 DEBUG:teuthology.orchestra.run.smithi083:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T09:15:00.996 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi158.front.sepia.ceph.com 2024-02-10T09:15:00.996 DEBUG:teuthology.orchestra.run.smithi158:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T09:15:01.176 DEBUG:teuthology.parallel:result is None 2024-02-10T09:15:01.188 DEBUG:teuthology.parallel:result is None 2024-02-10T09:15:01.188 INFO:teuthology.run_tasks:Running task install... 2024-02-10T09:15:01.196 DEBUG:teuthology.task.install:project ceph 2024-02-10T09:15:01.196 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-10T09:15:01.196 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-10T09:15:01.196 INFO:teuthology.task.install:Using flavor: default 2024-02-10T09:15:01.203 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T09:15:01.203 INFO:teuthology.task.install:extra packages: [] 2024-02-10T09:15:01.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-10T09:15:01.204 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T09:15:01.204 INFO:teuthology.packaging:ref: None 2024-02-10T09:15:01.204 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T09:15:01.204 INFO:teuthology.packaging:branch: None 2024-02-10T09:15:01.204 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:15:01.416 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T09:15:01.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T09:15:01.419 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-10T09:15:01.419 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T09:15:01.419 INFO:teuthology.packaging:ref: None 2024-02-10T09:15:01.419 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T09:15:01.419 INFO:teuthology.packaging:branch: None 2024-02-10T09:15:01.419 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:15:01.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T09:15:01.626 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-10T09:15:01.627 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-10T09:15:01.639 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-10T09:15:01.640 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-10T09:15:01.669 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T09:15:01.669 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:15:01.669 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T09:15:01.671 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T09:15:01.672 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:15:01.672 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T09:15:01.699 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T09:15:01.700 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T09:15:01.700 INFO:teuthology.packaging:ref: None 2024-02-10T09:15:01.700 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T09:15:01.700 INFO:teuthology.packaging:branch: None 2024-02-10T09:15:01.700 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:15:01.700 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T09:15:01.701 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T09:15:01.701 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T09:15:01.701 INFO:teuthology.packaging:ref: None 2024-02-10T09:15:01.701 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T09:15:01.702 INFO:teuthology.packaging:branch: None 2024-02-10T09:15:01.702 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:15:01.702 DEBUG:teuthology.orchestra.run.smithi158:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T09:15:01.768 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T09:15:01.769 DEBUG:teuthology.orchestra.run.smithi158:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T09:15:01.844 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T09:15:01.846 DEBUG:teuthology.orchestra.run.smithi158:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T09:15:01.910 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-02-10T09:15:01.911 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-02-10T09:15:01.913 INFO:teuthology.orchestra.run.smithi158.stdout:check_obsoletes = 1 2024-02-10T09:15:01.914 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum clean all 2024-02-10T09:15:02.179 INFO:teuthology.orchestra.run.smithi083.stdout:76 files removed 2024-02-10T09:15:02.184 INFO:teuthology.orchestra.run.smithi158.stdout:76 files removed 2024-02-10T09:15:02.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-02-10T09:15:02.211 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-radosgw 2024-02-10T09:15:02.687 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 939 kB/s | 74 kB 00:00 2024-02-10T09:15:02.699 INFO:teuthology.orchestra.run.smithi158.stdout:ceph packages for x86_64 938 kB/s | 74 kB 00:00 2024-02-10T09:15:02.804 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 194 kB/s | 15 kB 00:00 2024-02-10T09:15:02.816 INFO:teuthology.orchestra.run.smithi158.stdout:ceph noarch packages 193 kB/s | 15 kB 00:00 2024-02-10T09:15:02.897 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 28 kB/s | 1.7 kB 00:00 2024-02-10T09:15:02.942 INFO:teuthology.orchestra.run.smithi158.stdout:ceph source packages 18 kB/s | 1.7 kB 00:00 2024-02-10T09:15:09.351 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - AppStream 4.2 MB/s | 27 MB 00:06 2024-02-10T09:15:15.301 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - BaseOS 8.8 MB/s | 10 MB 00:01 2024-02-10T09:15:17.135 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras 76 kB/s | 18 kB 00:00 2024-02-10T09:15:17.494 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - Extras common packages 28 kB/s | 6.9 kB 00:00 2024-02-10T09:15:23.279 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - AppStream 1.3 MB/s | 27 MB 00:20 2024-02-10T09:15:29.238 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - BaseOS 9.5 MB/s | 10 MB 00:01 2024-02-10T09:15:31.191 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras 54 kB/s | 18 kB 00:00 2024-02-10T09:15:31.658 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-10T09:17:38.586 INFO:teuthology.orchestra.run.smithi158.stdout:CentOS Stream 8 - PowerTools 35 kB/s | 4.8 MB 02:20 2024-02-10T09:17:39.686 INFO:teuthology.orchestra.run.smithi158.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-02-10T09:17:41.203 INFO:teuthology.orchestra.run.smithi158.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-10T09:17:44.987 INFO:teuthology.orchestra.run.smithi158.stdout:lab-extras 376 kB/s | 24 kB 00:00 2024-02-10T09:17:48.458 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrading: 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:17:48.461 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-10T09:17:48.462 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout:Install 25 Packages 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout:Upgrade 2 Packages 2024-02-10T09:17:48.463 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:17:48.464 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 59 M 2024-02-10T09:17:48.464 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:17:49.196 INFO:teuthology.orchestra.run.smithi158.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 36 MB/s | 6.0 MB 00:00 2024-02-10T09:17:49.252 INFO:teuthology.orchestra.run.smithi158.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 443 kB/s | 25 kB 00:00 2024-02-10T09:17:49.296 INFO:teuthology.orchestra.run.smithi158.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 18 MB/s | 801 kB 00:00 2024-02-10T09:17:49.330 INFO:teuthology.orchestra.run.smithi158.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-10T09:17:49.438 INFO:teuthology.orchestra.run.smithi158.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2024-02-10T09:17:49.464 INFO:teuthology.orchestra.run.smithi158.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-10T09:17:49.489 INFO:teuthology.orchestra.run.smithi158.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-10T09:17:49.515 INFO:teuthology.orchestra.run.smithi158.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-10T09:17:49.548 INFO:teuthology.orchestra.run.smithi158.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-10T09:17:49.590 INFO:teuthology.orchestra.run.smithi158.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 8.6 MB/s | 368 kB 00:00 2024-02-10T09:17:49.616 INFO:teuthology.orchestra.run.smithi158.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 114 kB 00:00 2024-02-10T09:17:49.766 INFO:teuthology.orchestra.run.smithi158.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2024-02-10T09:17:50.066 INFO:teuthology.orchestra.run.smithi158.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2024-02-10T09:17:50.433 INFO:teuthology.orchestra.run.smithi158.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 709 kB/s | 259 kB 00:00 2024-02-10T09:17:50.458 INFO:teuthology.orchestra.run.smithi158.stdout:(15/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 172 kB/s | 145 kB 00:00 2024-02-10T09:17:50.492 INFO:teuthology.orchestra.run.smithi158.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 933 kB/s | 677 kB 00:00 2024-02-10T09:17:50.584 INFO:teuthology.orchestra.run.smithi158.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 292 kB/s | 44 kB 00:00 2024-02-10T09:17:50.834 INFO:teuthology.orchestra.run.smithi158.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 123 kB/s | 42 kB 00:00 2024-02-10T09:17:50.859 INFO:teuthology.orchestra.run.smithi158.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 128 kB/s | 47 kB 00:00 2024-02-10T09:17:50.901 INFO:teuthology.orchestra.run.smithi158.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 253 kB/s | 80 kB 00:00 2024-02-10T09:17:51.193 INFO:teuthology.orchestra.run.smithi158.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 324 kB/s | 116 kB 00:00 2024-02-10T09:17:51.269 INFO:teuthology.orchestra.run.smithi158.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 493 kB/s | 181 kB 00:00 2024-02-10T09:17:51.294 INFO:teuthology.orchestra.run.smithi158.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 705 kB/s | 306 kB 00:00 2024-02-10T09:17:51.311 INFO:teuthology.orchestra.run.smithi158.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 502 kB/s | 59 kB 00:00 2024-02-10T09:17:51.361 INFO:teuthology.orchestra.run.smithi158.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 816 kB/s | 75 kB 00:00 2024-02-10T09:17:51.444 INFO:teuthology.orchestra.run.smithi158.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2024-02-10T09:17:51.503 INFO:teuthology.orchestra.run.smithi158.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-02-10T09:17:51.508 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:17:51.508 INFO:teuthology.orchestra.run.smithi158.stdout:Total 19 MB/s | 59 MB 00:03 2024-02-10T09:17:51.578 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:17:51.627 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:17:51.627 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:17:51.982 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:17:51.985 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:17:52.137 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 8 - PowerTools 35 kB/s | 4.8 MB 02:20 2024-02-10T09:17:53.121 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:17:53.204 INFO:teuthology.orchestra.run.smithi083.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-02-10T09:17:53.326 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-10T09:17:53.541 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-10T09:17:53.728 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T09:17:54.145 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T09:17:54.344 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T09:17:54.445 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T09:17:54.678 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T09:17:54.853 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T09:17:54.915 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T09:17:54.949 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 16 MB 00:01 2024-02-10T09:17:55.179 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-10T09:17:55.729 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-10T09:17:55.886 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T09:17:55.970 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T09:17:56.149 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-10T09:17:56.353 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T09:17:56.421 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T09:17:56.583 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T09:17:56.753 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T09:17:56.928 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T09:17:57.005 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T09:17:57.200 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T09:17:57.619 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T09:17:57.836 INFO:teuthology.orchestra.run.smithi158.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T09:17:57.930 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T09:17:58.107 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-10T09:17:58.321 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-10T09:17:58.502 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T09:17:58.573 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T09:17:58.697 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 311 kB/s | 24 kB 00:00 2024-02-10T09:17:58.761 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-10T09:17:59.011 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-10T09:17:59.210 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-10T09:17:59.428 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-10T09:18:01.987 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T09:18:02.188 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:18:02.191 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:02.191 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:18:02.191 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:02.191 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-10T09:18:02.192 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-10T09:18:02.193 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout:Install 25 Packages 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 59 M 2024-02-10T09:18:02.194 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:18:02.227 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T09:18:02.648 INFO:teuthology.orchestra.run.smithi083.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 47 MB/s | 6.0 MB 00:00 2024-02-10T09:18:02.673 INFO:teuthology.orchestra.run.smithi083.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2024-02-10T09:18:02.706 INFO:teuthology.orchestra.run.smithi083.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-02-10T09:18:02.740 INFO:teuthology.orchestra.run.smithi083.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-10T09:18:02.824 INFO:teuthology.orchestra.run.smithi083.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 49 MB/s | 4.1 MB 00:00 2024-02-10T09:18:02.849 INFO:teuthology.orchestra.run.smithi083.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2024-02-10T09:18:02.875 INFO:teuthology.orchestra.run.smithi083.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-10T09:18:02.900 INFO:teuthology.orchestra.run.smithi083.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-10T09:18:02.923 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T09:18:02.925 INFO:teuthology.orchestra.run.smithi083.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-02-10T09:18:02.950 INFO:teuthology.orchestra.run.smithi083.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-02-10T09:18:02.976 INFO:teuthology.orchestra.run.smithi083.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-10T09:18:03.101 INFO:teuthology.orchestra.run.smithi083.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 22 MB/s | 13 MB 00:00 2024-02-10T09:18:03.199 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T09:18:03.304 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T09:18:03.368 INFO:teuthology.orchestra.run.smithi083.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 28 MB/s | 24 MB 00:00 2024-02-10T09:18:03.393 INFO:teuthology.orchestra.run.smithi083.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 348 kB/s | 145 kB 00:00 2024-02-10T09:18:03.452 INFO:teuthology.orchestra.run.smithi083.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 754 kB/s | 44 kB 00:00 2024-02-10T09:18:03.476 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T09:18:03.560 INFO:teuthology.orchestra.run.smithi083.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.4 MB/s | 677 kB 00:00 2024-02-10T09:18:03.719 INFO:teuthology.orchestra.run.smithi083.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 740 kB/s | 259 kB 00:00 2024-02-10T09:18:03.769 INFO:teuthology.orchestra.run.smithi083.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 133 kB/s | 42 kB 00:00 2024-02-10T09:18:03.886 INFO:teuthology.orchestra.run.smithi083.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 145 kB/s | 47 kB 00:00 2024-02-10T09:18:04.045 INFO:teuthology.orchestra.run.smithi083.stdout:(20/27): fmt-6.2.1-1.el8.x86_64.rpm 423 kB/s | 116 kB 00:00 2024-02-10T09:18:04.070 INFO:teuthology.orchestra.run.smithi083.stdout:(21/27): ndctl-libs-71.1-7.el8.x86_64.rpm 228 kB/s | 80 kB 00:00 2024-02-10T09:18:04.171 INFO:teuthology.orchestra.run.smithi083.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.4 MB/s | 181 kB 00:00 2024-02-10T09:18:04.196 INFO:teuthology.orchestra.run.smithi083.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 989 kB/s | 306 kB 00:00 2024-02-10T09:18:04.272 INFO:teuthology.orchestra.run.smithi083.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2024-02-10T09:18:04.297 INFO:teuthology.orchestra.run.smithi083.stdout:(25/27): liboath-2.6.2-3.el8.x86_64.rpm 260 kB/s | 59 kB 00:00 2024-02-10T09:18:04.322 INFO:teuthology.orchestra.run.smithi083.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 498 kB/s | 75 kB 00:00 2024-02-10T09:18:04.397 INFO:teuthology.orchestra.run.smithi083.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 32 MB/s | 4.0 MB 00:00 2024-02-10T09:18:04.399 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:18:04.399 INFO:teuthology.orchestra.run.smithi083.stdout:Total 27 MB/s | 59 MB 00:02 2024-02-10T09:18:04.468 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:18:04.516 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:18:04.517 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:18:04.870 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:18:04.873 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:18:06.006 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:18:06.241 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-10T09:18:06.443 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-10T09:18:06.624 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T09:18:07.011 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T09:18:07.199 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T09:18:07.306 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T09:18:07.525 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T09:18:07.674 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T09:18:07.763 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T09:18:08.000 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-10T09:18:08.532 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-10T09:18:08.682 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T09:18:08.758 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T09:18:08.920 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-10T09:18:09.107 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T09:18:09.157 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T09:18:09.371 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T09:18:09.515 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T09:18:09.691 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T09:18:09.767 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T09:18:09.944 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T09:18:10.380 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T09:18:10.557 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T09:18:10.649 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T09:18:10.843 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-10T09:18:11.050 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-10T09:18:11.190 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T09:18:11.268 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T09:18:11.474 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-10T09:18:11.690 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-10T09:18:11.855 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-10T09:18:12.049 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-10T09:18:14.632 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T09:18:14.874 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T09:18:15.567 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T09:18:15.888 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T09:18:15.996 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T09:18:16.122 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T09:18:22.884 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /sys 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /proc 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /run 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /mnt 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /var/tmp 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /home 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /tmp 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout:skipping the directory /dev 2024-02-10T09:18:22.885 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:24.245 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T09:18:24.320 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T09:18:24.320 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T09:18:24.525 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T09:18:24.525 INFO:teuthology.orchestra.run.smithi158.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T09:18:25.218 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-10T09:18:25.219 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T09:18:25.220 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-10T09:18:25.221 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout:Upgraded: 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.212 INFO:teuthology.orchestra.run.smithi158.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: leveldb-1.22-1.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.213 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-10T09:18:26.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.214 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:26.214 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:26.214 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:18:26.502 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-test 2024-02-10T09:18:27.261 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:43 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:18:28.780 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:18:28.781 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:18:28.781 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:18:28.781 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:18:28.781 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:18:28.781 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2024-02-10T09:18:28.782 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:28.783 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 53 M 2024-02-10T09:18:28.783 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 229 M 2024-02-10T09:18:28.783 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:18:29.049 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 169 kB 00:00 2024-02-10T09:18:29.723 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 69 MB/s | 52 MB 00:00 2024-02-10T09:18:29.781 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 257 kB/s | 188 kB 00:00 2024-02-10T09:18:29.807 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 244 kB/s | 203 kB 00:00 2024-02-10T09:18:30.007 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 314 kB/s | 71 kB 00:00 2024-02-10T09:18:30.199 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 679 kB/s | 323 kB 00:00 2024-02-10T09:18:30.200 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:18:30.200 INFO:teuthology.orchestra.run.smithi158.stdout:Total 37 MB/s | 53 MB 00:01 2024-02-10T09:18:30.242 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:18:30.263 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:18:30.263 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:18:30.582 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:18:30.585 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:18:31.071 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:18:31.325 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-10T09:18:31.517 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-10T09:18:31.680 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T09:18:31.761 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T09:18:31.962 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-10T09:18:32.105 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T09:18:35.290 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T09:18:35.290 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /dev 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /run 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /dev 2024-02-10T09:18:35.291 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T09:18:36.701 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T09:18:36.701 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T09:18:36.905 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T09:18:36.906 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T09:18:36.933 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T09:18:37.178 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T09:18:37.473 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T09:18:37.474 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-10T09:18:37.474 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-10T09:18:37.474 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-10T09:18:37.474 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-10T09:18:37.474 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-10T09:18:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T09:18:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-10T09:18:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-10T09:18:37.587 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-10T09:18:37.588 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-10T09:18:37.589 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:37.911 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:18:38.110 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.395 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: leveldb-1.22-1.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-10T09:18:38.396 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:38.397 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:18:38.710 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-02-10T09:18:38.875 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:00:54 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:18:39.463 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:41 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:18:40.381 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-10T09:18:40.385 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-10T09:18:40.386 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-10T09:18:40.387 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:Install 32 Packages 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 31 M 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 111 M 2024-02-10T09:18:40.388 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:18:40.976 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:18:40.977 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:40.978 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-02-10T09:18:40.979 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:40.979 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 53 M 2024-02-10T09:18:40.979 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 229 M 2024-02-10T09:18:40.979 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:18:41.168 INFO:teuthology.orchestra.run.smithi158.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 122 kB/s | 6.3 kB 00:00 2024-02-10T09:18:41.218 INFO:teuthology.orchestra.run.smithi158.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-02-10T09:18:41.251 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 169 kB 00:00 2024-02-10T09:18:41.277 INFO:teuthology.orchestra.run.smithi158.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-02-10T09:18:41.302 INFO:teuthology.orchestra.run.smithi158.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-10T09:18:41.386 INFO:teuthology.orchestra.run.smithi158.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-02-10T09:18:41.644 INFO:teuthology.orchestra.run.smithi158.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-02-10T09:18:41.669 INFO:teuthology.orchestra.run.smithi158.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2024-02-10T09:18:41.695 INFO:teuthology.orchestra.run.smithi158.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 400 kB/s | 157 kB 00:00 2024-02-10T09:18:41.787 INFO:teuthology.orchestra.run.smithi158.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 487 kB/s | 57 kB 00:00 2024-02-10T09:18:41.879 INFO:teuthology.orchestra.run.smithi158.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 442 kB/s | 103 kB 00:00 2024-02-10T09:18:41.893 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 74 MB/s | 52 MB 00:00 2024-02-10T09:18:41.912 INFO:teuthology.orchestra.run.smithi158.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.1 MB/s | 457 kB 00:00 2024-02-10T09:18:41.918 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 278 kB/s | 203 kB 00:00 2024-02-10T09:18:41.943 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 272 kB/s | 188 kB 00:00 2024-02-10T09:18:42.060 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2024-02-10T09:18:42.086 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 423 kB/s | 71 kB 00:00 2024-02-10T09:18:42.087 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:18:42.087 INFO:teuthology.orchestra.run.smithi083.stdout:Total 48 MB/s | 53 MB 00:01 2024-02-10T09:18:42.129 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:18:42.150 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:18:42.150 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:18:42.271 INFO:teuthology.orchestra.run.smithi158.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 514 kB/s | 249 kB 00:00 2024-02-10T09:18:42.296 INFO:teuthology.orchestra.run.smithi158.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 284 kB/s | 109 kB 00:00 2024-02-10T09:18:42.463 INFO:teuthology.orchestra.run.smithi158.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-02-10T09:18:42.472 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:18:42.475 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:18:42.530 INFO:teuthology.orchestra.run.smithi158.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 169 kB/s | 44 kB 00:00 2024-02-10T09:18:42.655 INFO:teuthology.orchestra.run.smithi158.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 516 kB/s | 185 kB 00:00 2024-02-10T09:18:42.731 INFO:teuthology.orchestra.run.smithi158.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2024-02-10T09:18:42.797 INFO:teuthology.orchestra.run.smithi158.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2024-02-10T09:18:42.823 INFO:teuthology.orchestra.run.smithi158.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 192 kB/s | 18 kB 00:00 2024-02-10T09:18:42.848 INFO:teuthology.orchestra.run.smithi158.stdout:(20/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 449 kB/s | 173 kB 00:00 2024-02-10T09:18:42.865 INFO:teuthology.orchestra.run.smithi158.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 725 kB/s | 49 kB 00:00 2024-02-10T09:18:42.899 INFO:teuthology.orchestra.run.smithi158.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 776 kB/s | 59 kB 00:00 2024-02-10T09:18:42.932 INFO:teuthology.orchestra.run.smithi158.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2024-02-10T09:18:42.966 INFO:teuthology.orchestra.run.smithi158.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 280 kB/s | 19 kB 00:00 2024-02-10T09:18:42.978 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:18:42.991 INFO:teuthology.orchestra.run.smithi158.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2024-02-10T09:18:43.016 INFO:teuthology.orchestra.run.smithi158.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2024-02-10T09:18:43.033 INFO:teuthology.orchestra.run.smithi158.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 435 kB/s | 29 kB 00:00 2024-02-10T09:18:43.067 INFO:teuthology.orchestra.run.smithi158.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-02-10T09:18:43.109 INFO:teuthology.orchestra.run.smithi158.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2024-02-10T09:18:43.167 INFO:teuthology.orchestra.run.smithi158.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 853 kB/s | 85 kB 00:00 2024-02-10T09:18:43.193 INFO:teuthology.orchestra.run.smithi158.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-10T09:18:43.197 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-10T09:18:43.218 INFO:teuthology.orchestra.run.smithi158.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 211 kB/s | 23 kB 00:00 2024-02-10T09:18:43.222 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:18:43.222 INFO:teuthology.orchestra.run.smithi158.stdout:Total 11 MB/s | 31 MB 00:02 2024-02-10T09:18:43.302 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:18:43.341 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:18:43.341 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:18:43.409 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-10T09:18:43.544 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T09:18:43.607 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T09:18:43.653 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:18:43.656 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:18:43.795 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-10T09:18:43.952 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T09:18:44.792 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:18:44.999 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-10T09:18:45.194 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-10T09:18:45.502 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-10T09:18:45.734 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-10T09:18:45.915 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-10T09:18:46.137 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-10T09:18:46.306 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-10T09:18:46.475 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-10T09:18:46.660 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-10T09:18:46.880 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-10T09:18:47.093 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-10T09:18:47.335 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-10T09:18:47.739 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-10T09:18:47.997 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-10T09:18:48.194 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-10T09:18:48.438 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-10T09:18:48.646 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T09:18:48.809 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T09:18:48.984 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-10T09:18:49.033 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T09:18:49.183 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-10T09:18:49.318 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T09:18:49.318 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-10T09:18:49.318 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-10T09:18:49.319 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-10T09:18:49.319 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-10T09:18:49.319 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-10T09:18:49.372 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-10T09:18:49.606 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:49.732 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:18:49.855 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-10T09:18:49.943 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-02-10T09:18:50.073 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-10T09:18:50.324 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-10T09:18:50.651 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-10T09:18:50.701 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:52 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:18:50.966 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-10T09:18:51.344 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-10T09:18:51.540 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T09:18:52.221 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-10T09:18:52.225 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-10T09:18:52.226 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout:Install 32 Packages 2024-02-10T09:18:52.227 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:18:52.228 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 31 M 2024-02-10T09:18:52.228 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 111 M 2024-02-10T09:18:52.228 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:18:52.580 INFO:teuthology.orchestra.run.smithi083.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 70 kB/s | 6.3 kB 00:00 2024-02-10T09:18:52.630 INFO:teuthology.orchestra.run.smithi083.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-02-10T09:18:52.688 INFO:teuthology.orchestra.run.smithi083.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-02-10T09:18:52.722 INFO:teuthology.orchestra.run.smithi083.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-10T09:18:52.798 INFO:teuthology.orchestra.run.smithi083.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-02-10T09:18:53.031 INFO:teuthology.orchestra.run.smithi083.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-02-10T09:18:53.056 INFO:teuthology.orchestra.run.smithi083.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 150 kB/s | 39 kB 00:00 2024-02-10T09:18:53.068 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T09:18:53.081 INFO:teuthology.orchestra.run.smithi083.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 437 kB/s | 157 kB 00:00 2024-02-10T09:18:53.149 INFO:teuthology.orchestra.run.smithi083.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 619 kB/s | 57 kB 00:00 2024-02-10T09:18:53.249 INFO:teuthology.orchestra.run.smithi083.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 476 kB/s | 103 kB 00:00 2024-02-10T09:18:53.273 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T09:18:53.274 INFO:teuthology.orchestra.run.smithi083.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.3 MB/s | 457 kB 00:00 2024-02-10T09:18:53.641 INFO:teuthology.orchestra.run.smithi083.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 506 kB/s | 249 kB 00:00 2024-02-10T09:18:53.666 INFO:teuthology.orchestra.run.smithi083.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 278 kB/s | 109 kB 00:00 2024-02-10T09:18:53.707 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T09:18:53.842 INFO:teuthology.orchestra.run.smithi083.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.1 MB/s | 641 kB 00:00 2024-02-10T09:18:53.940 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T09:18:53.992 INFO:teuthology.orchestra.run.smithi083.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 124 kB/s | 44 kB 00:00 2024-02-10T09:18:54.134 INFO:teuthology.orchestra.run.smithi083.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 396 kB/s | 185 kB 00:00 2024-02-10T09:18:54.162 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T09:18:54.243 INFO:teuthology.orchestra.run.smithi083.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-02-10T09:18:54.293 INFO:teuthology.orchestra.run.smithi083.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 383 kB/s | 173 kB 00:00 2024-02-10T09:18:54.326 INFO:teuthology.orchestra.run.smithi083.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-02-10T09:18:54.343 INFO:teuthology.orchestra.run.smithi083.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-02-10T09:18:54.360 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T09:18:54.378 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T09:18:54.402 INFO:teuthology.orchestra.run.smithi083.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 447 kB/s | 49 kB 00:00 2024-02-10T09:18:54.427 INFO:teuthology.orchestra.run.smithi083.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 585 kB/s | 59 kB 00:00 2024-02-10T09:18:54.494 INFO:teuthology.orchestra.run.smithi083.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2024-02-10T09:18:54.519 INFO:teuthology.orchestra.run.smithi083.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 203 kB/s | 19 kB 00:00 2024-02-10T09:18:54.534 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T09:18:54.545 INFO:teuthology.orchestra.run.smithi083.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-02-10T09:18:54.587 INFO:teuthology.orchestra.run.smithi083.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 259 kB/s | 24 kB 00:00 2024-02-10T09:18:54.612 INFO:teuthology.orchestra.run.smithi083.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 317 kB/s | 29 kB 00:00 2024-02-10T09:18:54.637 INFO:teuthology.orchestra.run.smithi083.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 298 kB/s | 27 kB 00:00 2024-02-10T09:18:54.746 INFO:teuthology.orchestra.run.smithi083.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-02-10T09:18:54.771 INFO:teuthology.orchestra.run.smithi083.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 639 kB/s | 85 kB 00:00 2024-02-10T09:18:54.796 INFO:teuthology.orchestra.run.smithi083.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2024-02-10T09:18:54.888 INFO:teuthology.orchestra.run.smithi083.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 161 kB/s | 23 kB 00:00 2024-02-10T09:18:54.890 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:18:54.890 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 31 MB 00:02 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-10T09:18:54.934 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-10T09:18:54.935 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-10T09:18:54.936 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-10T09:18:54.936 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-10T09:18:54.936 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-10T09:18:54.936 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-10T09:18:54.936 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-10T09:18:54.936 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-10T09:18:54.969 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:18:55.007 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:18:55.007 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:18:55.321 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:18:55.324 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-10T09:18:55.751 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-portend-2.6-1.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-10T09:18:55.752 INFO:teuthology.orchestra.run.smithi158.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:18:55.753 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:18:56.046 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-base 2024-02-10T09:18:56.428 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:18:56.631 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-10T09:18:56.813 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:12 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:18:56.850 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-10T09:18:57.083 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-10T09:18:57.315 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-10T09:18:57.488 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-10T09:18:57.660 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-10T09:18:57.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-10T09:18:58.031 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-10T09:18:58.208 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-10T09:18:58.309 INFO:teuthology.orchestra.run.smithi158.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:18:58.354 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:18:58.356 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:18:58.356 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:18:58.404 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-10T09:18:58.472 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install cephadm 2024-02-10T09:18:58.624 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-10T09:18:58.905 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-10T09:18:59.219 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:15 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:18:59.273 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-10T09:18:59.553 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-10T09:18:59.725 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-10T09:18:59.982 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-10T09:19:00.167 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T09:19:00.485 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-10T09:19:00.664 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-10T09:19:00.749 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:00.749 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:00.749 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:19:00.749 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:00.749 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 71 k 2024-02-10T09:19:00.750 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 290 k 2024-02-10T09:19:00.751 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:00.790 INFO:teuthology.orchestra.run.smithi158.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 71 kB 00:00 2024-02-10T09:19:00.791 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:00.791 INFO:teuthology.orchestra.run.smithi158.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-10T09:19:00.791 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:00.798 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:00.798 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:00.800 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:00.803 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:00.835 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-10T09:19:00.847 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:01.076 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-10T09:19:01.321 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-10T09:19:01.343 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:01.543 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:01.568 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-10T09:19:01.774 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:01.784 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-10T09:19:02.090 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-10T09:19:02.138 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:02.138 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:02.138 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:02.139 INFO:teuthology.orchestra.run.smithi158.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-10T09:19:02.139 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:02.139 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:02.372 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-10T09:19:02.380 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T09:19:02.786 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-10T09:19:03.006 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T09:19:03.152 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:19 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:04.544 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T09:19:04.675 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:04.675 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:04.675 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repo Size 2024-02-10T09:19:04.675 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:19:04.676 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:04.677 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 163 k 2024-02-10T09:19:04.677 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 587 k 2024-02-10T09:19:04.677 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:04.723 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T09:19:04.737 INFO:teuthology.orchestra.run.smithi158.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.6 MB/s | 163 kB 00:00 2024-02-10T09:19:04.738 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:04.738 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.6 MB/s | 163 kB 00:00 2024-02-10T09:19:04.738 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:04.749 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:04.749 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:04.796 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:04.799 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:04.920 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:05.141 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:05.156 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T09:19:05.356 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T09:19:05.558 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:05.578 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T09:19:05.792 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T09:19:05.809 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T09:19:05.929 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:05.929 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:05.929 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:05.929 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:05.930 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:05.930 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:05.966 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T09:19:06.154 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr 2024-02-10T09:19:06.374 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-10T09:19:06.375 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-10T09:19:06.376 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-10T09:19:06.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-10T09:19:06.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-10T09:19:06.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-10T09:19:06.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-10T09:19:06.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-10T09:19:06.377 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-10T09:19:06.378 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-10T09:19:06.378 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-10T09:19:06.916 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:22 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-10T09:19:07.172 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-10T09:19:07.173 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-2.6-1.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-10T09:19:07.174 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-10T09:19:07.175 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-10T09:19:07.175 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:07.175 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:07.490 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-02-10T09:19:08.245 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:10 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:08.395 INFO:teuthology.orchestra.run.smithi158.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:19:08.439 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:08.440 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:19:08.440 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:08.578 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T09:19:09.335 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:25 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:09.713 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:19:09.759 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:09.760 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:19:09.760 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:09.884 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-02-10T09:19:10.636 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:12 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:10.843 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-10T09:19:10.844 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 3.0 M 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 15 M 2024-02-10T09:19:10.845 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:11.274 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 137 kB/s | 8.9 kB 00:00 2024-02-10T09:19:11.291 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 292 kB/s | 24 kB 00:00 2024-02-10T09:19:11.365 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-02-10T09:19:11.483 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2024-02-10T09:19:11.608 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 127 kB/s | 43 kB 00:00 2024-02-10T09:19:11.633 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 732 kB/s | 196 kB 00:00 2024-02-10T09:19:11.635 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:11.636 INFO:teuthology.orchestra.run.smithi158.stdout:Total 3.7 MB/s | 3.0 MB 00:00 2024-02-10T09:19:11.645 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:11.655 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:11.655 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:11.810 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:11.814 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:12.080 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:12.177 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:12.178 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:12.178 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 71 k 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 290 k 2024-02-10T09:19:12.179 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:19:12.220 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-02-10T09:19:12.221 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:12.221 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-10T09:19:12.221 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:19:12.228 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:19:12.228 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:19:12.229 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:19:12.232 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:19:12.277 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:19:12.304 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-10T09:19:12.557 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-10T09:19:12.785 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:12.792 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-10T09:19:12.904 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-10T09:19:12.951 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:13.177 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:13.564 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T09:19:13.564 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:13.564 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:13.564 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-10T09:19:13.564 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:13.564 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:13.798 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T09:19:13.887 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-10T09:19:14.194 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-10T09:19:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-10T09:19:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-10T09:19:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-10T09:19:14.491 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-10T09:19:14.554 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:16 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-10T09:19:14.868 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:14.869 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:15.008 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T09:19:15.759 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:31 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:16.048 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:16.049 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:16.049 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:19:16.049 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:16.049 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 163 k 2024-02-10T09:19:16.049 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 587 k 2024-02-10T09:19:16.049 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:19:16.485 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 374 kB/s | 163 kB 00:00 2024-02-10T09:19:16.485 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:16.486 INFO:teuthology.orchestra.run.smithi083.stdout:Total 373 kB/s | 163 kB 00:00 2024-02-10T09:19:16.486 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:19:16.498 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:19:16.498 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:19:16.550 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:19:16.553 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:19:16.677 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:19:16.856 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:17.270 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:17.295 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-10T09:19:17.297 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout:============================================================================================ 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout:Install 8 Packages 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:17.298 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 37 M 2024-02-10T09:19:17.299 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 210 M 2024-02-10T09:19:17.299 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:17.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:17.660 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:17.660 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:17.660 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:17.660 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:17.660 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:17.710 INFO:teuthology.orchestra.run.smithi158.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 48 MB/s | 8.6 MB 00:00 2024-02-10T09:19:17.894 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-02-10T09:19:18.105 INFO:teuthology.orchestra.run.smithi158.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.2 MB/s | 4.7 MB 00:00 2024-02-10T09:19:18.172 INFO:teuthology.orchestra.run.smithi158.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.6 MB/s | 4.9 MB 00:00 2024-02-10T09:19:18.197 INFO:teuthology.orchestra.run.smithi158.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-10T09:19:18.654 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:20 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:18.739 INFO:teuthology.orchestra.run.smithi158.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2024-02-10T09:19:18.839 INFO:teuthology.orchestra.run.smithi158.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2024-02-10T09:19:19.140 INFO:teuthology.orchestra.run.smithi158.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 15 MB/s | 14 MB 00:00 2024-02-10T09:19:20.123 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:19:20.140 INFO:teuthology.orchestra.run.smithi158.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.5 MB/s | 3.7 MB 00:02 2024-02-10T09:19:20.162 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:20.163 INFO:teuthology.orchestra.run.smithi158.stdout:Total 13 MB/s | 37 MB 00:02 2024-02-10T09:19:20.169 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:20.170 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:19:20.171 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:20.279 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T09:19:20.374 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:20.392 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:20.392 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:20.739 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:20.743 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:21.027 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:23 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:21.651 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:21.805 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T09:19:21.945 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T09:19:22.080 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T09:19:22.545 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:22.546 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:22.546 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:19:22.546 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:22.546 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:22.546 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-02-10T09:19:22.547 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:22.548 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.0 M 2024-02-10T09:19:22.548 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 15 M 2024-02-10T09:19:22.548 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:19:22.680 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T09:19:22.871 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T09:19:22.972 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 587 kB/s | 24 kB 00:00 2024-02-10T09:19:22.988 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 155 kB/s | 8.9 kB 00:00 2024-02-10T09:19:23.055 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2024-02-10T09:19:23.247 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 130 kB/s | 34 kB 00:00 2024-02-10T09:19:23.306 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 127 kB/s | 43 kB 00:00 2024-02-10T09:19:23.373 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 619 kB/s | 196 kB 00:00 2024-02-10T09:19:23.376 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:23.376 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2024-02-10T09:19:23.386 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:19:23.395 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:19:23.395 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:19:23.482 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T09:19:23.560 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:19:23.563 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:19:23.646 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T09:19:23.834 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:19:24.019 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-10T09:19:24.273 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-10T09:19:24.338 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T09:19:24.474 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-10T09:19:24.603 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-10T09:19:24.682 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T09:19:24.954 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T09:19:25.576 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-10T09:19:25.886 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:26.177 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:26.177 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-10T09:19:26.177 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-10T09:19:26.177 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-10T09:19:26.177 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-10T09:19:26.177 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-10T09:19:26.584 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-10T09:19:26.584 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:26.584 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:26.584 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-10T09:19:26.584 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-10T09:19:26.585 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-10T09:19:26.585 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-10T09:19:26.585 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-10T09:19:26.585 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-10T09:19:26.585 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:26.585 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:26.715 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T09:19:26.720 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T09:19:27.465 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:29 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:28.580 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-10T09:19:28.754 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T09:19:28.982 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:28.983 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-10T09:19:28.984 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout:Install 8 Packages 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 37 M 2024-02-10T09:19:28.985 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 210 M 2024-02-10T09:19:28.986 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:19:29.067 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T09:19:29.067 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-10T09:19:29.067 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-10T09:19:29.068 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-10T09:19:29.068 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-10T09:19:29.068 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T09:19:29.068 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-10T09:19:29.068 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-10T09:19:29.203 INFO:teuthology.orchestra.run.smithi083.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 63 MB/s | 8.6 MB 00:00 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-10T09:19:29.494 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-10T09:19:29.495 INFO:teuthology.orchestra.run.smithi158.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-10T09:19:29.495 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-10T09:19:29.495 INFO:teuthology.orchestra.run.smithi158.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-10T09:19:29.495 INFO:teuthology.orchestra.run.smithi158.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-10T09:19:29.495 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:29.495 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:29.609 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-rook 2024-02-10T09:19:29.662 INFO:teuthology.orchestra.run.smithi083.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.2 MB/s | 4.9 MB 00:00 2024-02-10T09:19:29.720 INFO:teuthology.orchestra.run.smithi083.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 7.2 MB/s | 4.7 MB 00:00 2024-02-10T09:19:29.754 INFO:teuthology.orchestra.run.smithi083.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-10T09:19:30.323 INFO:teuthology.orchestra.run.smithi083.stdout:(5/8): python3-scipy-1.0.0-21.module_el8+762+77 24 MB/s | 14 MB 00:00 2024-02-10T09:19:30.360 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:30.429 INFO:teuthology.orchestra.run.smithi083.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 956 kB/s | 645 kB 00:00 2024-02-10T09:19:30.722 INFO:teuthology.orchestra.run.smithi083.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 433 kB/s | 172 kB 00:00 2024-02-10T09:19:31.871 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-10T09:19:31.873 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout:Install 13 Packages 2024-02-10T09:19:31.874 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:31.875 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 1.9 M 2024-02-10T09:19:31.875 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 24 M 2024-02-10T09:19:31.875 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:32.096 INFO:teuthology.orchestra.run.smithi083.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.3 MB/s | 3.7 MB 00:02 2024-02-10T09:19:32.097 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:32.097 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 37 MB 00:03 2024-02-10T09:19:32.331 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:19:32.349 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:19:32.349 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:19:32.482 INFO:teuthology.orchestra.run.smithi158.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 620 kB/s | 27 kB 00:00 2024-02-10T09:19:32.621 INFO:teuthology.orchestra.run.smithi158.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 145 kB/s | 27 kB 00:00 2024-02-10T09:19:32.641 INFO:teuthology.orchestra.run.smithi158.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 99 kB/s | 20 kB 00:00 2024-02-10T09:19:32.662 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:19:32.665 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:19:32.808 INFO:teuthology.orchestra.run.smithi158.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 388 kB/s | 126 kB 00:00 2024-02-10T09:19:32.867 INFO:teuthology.orchestra.run.smithi158.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 449 kB/s | 110 kB 00:00 2024-02-10T09:19:33.117 INFO:teuthology.orchestra.run.smithi158.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 138 kB/s | 43 kB 00:00 2024-02-10T09:19:33.142 INFO:teuthology.orchestra.run.smithi158.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 309 kB/s | 155 kB 00:00 2024-02-10T09:19:33.243 INFO:teuthology.orchestra.run.smithi158.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 88 kB/s | 33 kB 00:00 2024-02-10T09:19:33.409 INFO:teuthology.orchestra.run.smithi158.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 56 kB/s | 16 kB 00:00 2024-02-10T09:19:33.526 INFO:teuthology.orchestra.run.smithi158.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 212 kB/s | 81 kB 00:00 2024-02-10T09:19:33.550 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:19:33.585 INFO:teuthology.orchestra.run.smithi158.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 366 kB/s | 64 kB 00:00 2024-02-10T09:19:33.627 INFO:teuthology.orchestra.run.smithi158.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 612 kB/s | 61 kB 00:00 2024-02-10T09:19:33.702 INFO:teuthology.orchestra.run.smithi158.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.5 MB/s | 1.2 MB 00:00 2024-02-10T09:19:33.705 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:33.705 INFO:teuthology.orchestra.run.smithi158.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2024-02-10T09:19:33.762 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:33.778 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:33.778 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:33.867 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:33.870 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:33.954 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T09:19:34.088 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T09:19:34.262 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T09:19:34.410 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:34.647 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-10T09:19:34.861 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T09:19:34.888 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-10T09:19:35.079 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T09:19:35.113 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-10T09:19:35.335 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-10T09:19:35.568 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-10T09:19:35.671 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T09:19:35.802 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-10T09:19:35.862 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T09:19:36.096 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-10T09:19:36.360 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-10T09:19:36.557 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T09:19:36.882 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T09:19:37.154 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T09:19:37.380 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-10T09:19:37.676 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-10T09:19:37.906 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-10T09:19:38.102 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-10T09:19:38.285 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T09:19:38.597 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-10T09:19:38.598 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-10T09:19:38.916 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T09:19:39.058 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-10T09:19:39.058 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-10T09:19:39.059 INFO:teuthology.orchestra.run.smithi158.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-10T09:19:39.060 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:39.060 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:39.173 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T09:19:39.930 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:01:55 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:40.260 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-10T09:19:40.478 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-10T09:19:40.800 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-10T09:19:41.236 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:41.237 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:41.344 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-02-10T09:19:41.451 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: Package Arch Version Repository Size 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-10T09:19:41.453 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout:Install 6 Packages 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 5.8 M 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 24 M 2024-02-10T09:19:41.454 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:41.826 INFO:teuthology.orchestra.run.smithi158.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-02-10T09:19:42.101 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:42.210 INFO:teuthology.orchestra.run.smithi158.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-02-10T09:19:42.535 INFO:teuthology.orchestra.run.smithi158.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.3 MB/s | 4.8 MB 00:00 2024-02-10T09:19:42.685 INFO:teuthology.orchestra.run.smithi158.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 25 kB/s | 21 kB 00:00 2024-02-10T09:19:43.536 INFO:teuthology.orchestra.run.smithi158.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 49 kB/s | 49 kB 00:01 2024-02-10T09:19:43.622 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:43.624 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:43.624 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:19:43.624 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:43.624 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:43.624 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-10T09:19:43.624 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:43.625 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:43.626 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:43.626 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-02-10T09:19:43.626 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:43.626 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 1.9 M 2024-02-10T09:19:43.626 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 24 M 2024-02-10T09:19:43.626 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:19:43.777 INFO:teuthology.orchestra.run.smithi158.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 227 kB/s | 355 kB 00:01 2024-02-10T09:19:43.780 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:43.780 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.5 MB/s | 5.8 MB 00:02 2024-02-10T09:19:43.834 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:43.844 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:43.844 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:43.953 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:43.957 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:44.226 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:44.381 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 705 kB/s | 27 kB 00:00 2024-02-10T09:19:44.477 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-10T09:19:44.481 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 144 kB/s | 20 kB 00:00 2024-02-10T09:19:44.498 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 171 kB/s | 27 kB 00:00 2024-02-10T09:19:44.582 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.1 MB/s | 110 kB 00:00 2024-02-10T09:19:44.607 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 560 kB/s | 126 kB 00:00 2024-02-10T09:19:44.691 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 395 kB/s | 33 kB 00:00 2024-02-10T09:19:44.732 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 393 kB/s | 16 kB 00:00 2024-02-10T09:19:44.788 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-10T09:19:44.808 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.1 MB/s | 81 kB 00:00 2024-02-10T09:19:44.900 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 134 kB/s | 43 kB 00:00 2024-02-10T09:19:44.917 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 370 kB/s | 155 kB 00:00 2024-02-10T09:19:44.981 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-kubernetes-11.0.0-6.el8.noarch 6.7 MB/s | 1.2 MB 00:00 2024-02-10T09:19:45.017 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 548 kB/s | 64 kB 00:00 2024-02-10T09:19:45.059 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-websocket-client-0.56.0-5.el8. 431 kB/s | 61 kB 00:00 2024-02-10T09:19:45.060 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:45.060 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2024-02-10T09:19:45.113 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:19:45.129 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:19:45.129 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:19:45.215 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:19:45.218 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:19:45.706 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-10T09:19:45.716 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:19:45.963 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-10T09:19:46.065 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-10T09:19:46.161 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-10T09:19:46.306 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-10T09:19:46.379 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-10T09:19:46.509 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:46.600 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-10T09:19:46.791 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-10T09:19:46.794 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:46.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-10T09:19:46.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-10T09:19:46.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-10T09:19:46.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-10T09:19:46.795 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-10T09:19:47.014 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-10T09:19:47.165 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-10T09:19:47.165 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:47.165 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:47.166 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-10T09:19:47.166 INFO:teuthology.orchestra.run.smithi158.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-10T09:19:47.166 INFO:teuthology.orchestra.run.smithi158.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-10T09:19:47.166 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:47.166 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:47.292 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-10T09:19:47.303 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install ceph-fuse 2024-02-10T09:19:47.532 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-10T09:19:48.058 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:04 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-10T09:19:48.822 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-10T09:19:49.054 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-10T09:19:49.235 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-10T09:19:49.418 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T09:19:49.585 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:Installing dependencies: 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout:Install 2 Packages 2024-02-10T09:19:49.586 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:49.587 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 968 k 2024-02-10T09:19:49.587 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 3.2 M 2024-02-10T09:19:49.587 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:49.699 INFO:teuthology.orchestra.run.smithi158.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-10T09:19:49.709 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-10T09:19:49.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-10T09:19:49.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-10T09:19:49.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-10T09:19:49.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-10T09:19:49.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-10T09:19:49.710 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-10T09:19:49.947 INFO:teuthology.orchestra.run.smithi158.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 261 kB/s | 83 kB 00:00 2024-02-10T09:19:49.947 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:49.948 INFO:teuthology.orchestra.run.smithi158.stdout:Total 2.6 MB/s | 968 kB 00:00 2024-02-10T09:19:49.958 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:49.978 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:49.978 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:50.065 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:50.068 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:50.156 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-10T09:19:50.157 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-10T09:19:50.158 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:50.158 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:50.241 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:50.282 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T09:19:50.500 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-10T09:19:50.684 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T09:19:51.032 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:53 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:51.083 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T09:19:51.083 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-10T09:19:51.474 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-10T09:19:51.474 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:51.474 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:51.474 INFO:teuthology.orchestra.run.smithi158.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-10T09:19:51.474 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:51.474 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:51.729 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librados-devel 2024-02-10T09:19:52.493 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:08 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:52.546 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-10T09:19:52.547 INFO:teuthology.orchestra.run.smithi083.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-10T09:19:52.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:52.548 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:52.548 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:52.548 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-02-10T09:19:52.548 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:52.548 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 5.8 M 2024-02-10T09:19:52.549 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 24 M 2024-02-10T09:19:52.549 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:19:52.709 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-02-10T09:19:53.084 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-02-10T09:19:53.134 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 50 kB/s | 21 kB 00:00 2024-02-10T09:19:53.326 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 255 kB/s | 49 kB 00:00 2024-02-10T09:19:53.918 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 426 kB/s | 355 kB 00:00 2024-02-10T09:19:54.005 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:54.006 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:54.006 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:19:54.006 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:54.006 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:19:54.006 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-10T09:19:54.006 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 131 k 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 439 k 2024-02-10T09:19:54.007 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:19:54.035 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 3.5 MB/s | 4.8 MB 00:01 2024-02-10T09:19:54.037 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:54.037 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.9 MB/s | 5.8 MB 00:01 2024-02-10T09:19:54.089 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:19:54.099 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:19:54.099 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:19:54.216 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:19:54.219 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:19:54.276 INFO:teuthology.orchestra.run.smithi158.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 488 kB/s | 131 kB 00:00 2024-02-10T09:19:54.277 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:19:54.277 INFO:teuthology.orchestra.run.smithi158.stdout:Total 486 kB/s | 131 kB 00:00 2024-02-10T09:19:54.278 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:19:54.283 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:19:54.283 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:19:54.336 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:19:54.340 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:19:54.547 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:19:54.783 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-10T09:19:54.787 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:19:55.015 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:55.078 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-10T09:19:55.309 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:55.685 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:19:55.686 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:55.686 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:19:55.686 INFO:teuthology.orchestra.run.smithi158.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-10T09:19:55.686 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:19:55.686 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:55.897 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install libcephfs2 2024-02-10T09:19:56.009 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-10T09:19:56.363 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-10T09:19:56.639 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-10T09:19:56.665 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:12 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:56.842 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:57.112 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T09:19:57.113 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-10T09:19:57.113 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-10T09:19:57.113 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-10T09:19:57.113 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-10T09:19:57.113 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-10T09:19:57.505 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-10T09:19:57.505 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:57.505 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:19:57.506 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-10T09:19:57.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-10T09:19:57.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-10T09:19:57.506 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:57.506 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:19:57.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-02-10T09:19:58.135 INFO:teuthology.orchestra.run.smithi158.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:19:58.180 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:19:58.182 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:19:58.182 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:19:58.282 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install libcephfs-devel 2024-02-10T09:19:58.389 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:00 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:19:59.031 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:15 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:19:59.889 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:19:59.889 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:59.889 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:19:59.889 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-02-10T09:19:59.890 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:19:59.891 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 968 k 2024-02-10T09:19:59.891 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 3.2 M 2024-02-10T09:19:59.891 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:20:00.016 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 15 MB/s | 885 kB 00:00 2024-02-10T09:20:00.274 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 263 kB/s | 83 kB 00:00 2024-02-10T09:20:00.274 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:00.275 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.5 MB/s | 968 kB 00:00 2024-02-10T09:20:00.284 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:20:00.304 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:20:00.305 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:20:00.391 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:20:00.394 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:20:00.539 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:00.539 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:20:00.540 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:00.541 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 24 k 2024-02-10T09:20:00.541 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 94 k 2024-02-10T09:20:00.541 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:20:00.577 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:20:00.833 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-10T09:20:00.932 INFO:teuthology.orchestra.run.smithi158.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 62 kB/s | 24 kB 00:00 2024-02-10T09:20:00.932 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:00.933 INFO:teuthology.orchestra.run.smithi158.stdout:Total 62 kB/s | 24 kB 00:00 2024-02-10T09:20:00.933 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:20:00.936 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:20:00.937 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:20:00.939 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:20:00.942 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:20:00.990 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:20:01.008 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T09:20:01.189 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T09:20:01.406 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-10T09:20:01.463 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:01.801 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-10T09:20:01.801 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:01.801 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:20:01.801 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-10T09:20:01.801 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:01.801 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:01.810 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:01.811 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:01.811 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:20:01.811 INFO:teuthology.orchestra.run.smithi158.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:01.811 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:01.811 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:01.929 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librados2 2024-02-10T09:20:02.053 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-02-10T09:20:02.692 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:18 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:02.815 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:04 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:04.180 INFO:teuthology.orchestra.run.smithi158.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:04.226 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:04.227 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:20:04.227 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:04.328 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install librbd1 2024-02-10T09:20:04.328 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:04.329 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 131 k 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 439 k 2024-02-10T09:20:04.330 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:20:04.379 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2024-02-10T09:20:04.380 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:04.380 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.6 MB/s | 131 kB 00:00 2024-02-10T09:20:04.380 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:20:04.385 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:20:04.385 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:20:04.435 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:20:04.439 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:20:04.562 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:20:05.085 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:21 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:05.683 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:05.973 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:06.353 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:06.353 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:06.354 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:20:06.354 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:06.354 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:06.354 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:06.557 INFO:teuthology.orchestra.run.smithi158.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:06.569 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-02-10T09:20:06.602 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:06.603 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:20:06.603 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:06.710 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rados 2024-02-10T09:20:07.327 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:09 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:07.461 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:23 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:08.791 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:08.837 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:08.838 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:08.838 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:08.930 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:08.954 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-02-10T09:20:08.976 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:08.977 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:20:08.977 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:09.085 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rgw 2024-02-10T09:20:09.703 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:11 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:09.838 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:25 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:11.205 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:11.207 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 24 k 2024-02-10T09:20:11.208 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 94 k 2024-02-10T09:20:11.208 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:20:11.292 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:11.340 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:11.362 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:20:11.362 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:11.543 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-cephfs 2024-02-10T09:20:11.861 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 37 kB/s | 24 kB 00:00 2024-02-10T09:20:11.862 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:11.862 INFO:teuthology.orchestra.run.smithi083.stdout:Total 37 kB/s | 24 kB 00:00 2024-02-10T09:20:11.863 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:20:11.867 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:20:11.867 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:20:11.870 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:20:11.873 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:20:12.047 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:20:12.248 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:12.300 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:28 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:12.530 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:12.927 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:12.928 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:20:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:12.928 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:12.928 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:13.065 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-02-10T09:20:13.761 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:13.807 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:13.808 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:20:13.808 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:13.818 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:15 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:13.917 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install python3-rbd 2024-02-10T09:20:14.682 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:30 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:15.290 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:15.336 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:15.338 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:15.338 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:15.454 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-02-10T09:20:16.160 INFO:teuthology.orchestra.run.smithi158.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:16.205 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:18 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:16.206 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:16.208 INFO:teuthology.orchestra.run.smithi158.stdout:Nothing to do. 2024-02-10T09:20:16.208 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:16.317 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-fuse 2024-02-10T09:20:17.071 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:33 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:17.677 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:17.722 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:17.724 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:17.724 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:17.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-02-10T09:20:18.573 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:18.573 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:18.573 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:18.573 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:18.573 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:18.574 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 89 k 2024-02-10T09:20:18.575 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 298 k 2024-02-10T09:20:18.575 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:20:18.587 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:20 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:18.622 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-02-10T09:20:18.622 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:18.622 INFO:teuthology.orchestra.run.smithi158.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-02-10T09:20:18.623 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:20:18.632 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:20:18.633 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:20:18.682 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:20:18.685 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:20:18.789 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:20:18.977 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:19.262 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:19.641 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:19.641 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:19.641 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:20:19.641 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:19.641 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:19.641 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:19.869 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-mirror 2024-02-10T09:20:20.069 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:20.114 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:20.115 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:20.115 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:20.232 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-02-10T09:20:20.626 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:36 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:20.978 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:22 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:22.127 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:20:22.128 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 3.7 M 2024-02-10T09:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 18 M 2024-02-10T09:20:22.129 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:20:22.250 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 31 MB/s | 3.7 MB 00:00 2024-02-10T09:20:22.250 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:22.251 INFO:teuthology.orchestra.run.smithi158.stdout:Total 30 MB/s | 3.7 MB 00:00 2024-02-10T09:20:22.251 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:20:22.263 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:20:22.263 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:20:22.326 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:20:22.329 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:20:22.441 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:22.486 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:22.488 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:22.488 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:22.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-02-10T09:20:22.806 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:20:23.012 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:23.419 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:25 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:23.422 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:23.798 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:23.798 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:23.799 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:20:23.799 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:23.799 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:23.799 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:24.013 DEBUG:teuthology.orchestra.run.smithi158:> sudo yum -y install rbd-nbd 2024-02-10T09:20:24.778 INFO:teuthology.orchestra.run.smithi158.stdout:Last metadata expiration check: 0:02:40 ago on Sat 10 Feb 2024 09:17:44 AM UTC. 2024-02-10T09:20:24.881 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:24.927 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:24.928 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:24.928 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:25.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-02-10T09:20:25.806 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:27 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:26.287 INFO:teuthology.orchestra.run.smithi158.stdout:Dependencies resolved. 2024-02-10T09:20:26.288 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:26.288 INFO:teuthology.orchestra.run.smithi158.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:Installing: 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction Summary 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:================================================================================ 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:Install 1 Package 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:Total download size: 173 k 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:Installed size: 649 k 2024-02-10T09:20:26.289 INFO:teuthology.orchestra.run.smithi158.stdout:Downloading Packages: 2024-02-10T09:20:26.344 INFO:teuthology.orchestra.run.smithi158.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 173 kB 00:00 2024-02-10T09:20:26.345 INFO:teuthology.orchestra.run.smithi158.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:26.345 INFO:teuthology.orchestra.run.smithi158.stdout:Total 3.0 MB/s | 173 kB 00:00 2024-02-10T09:20:26.345 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction check 2024-02-10T09:20:26.356 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction check succeeded. 2024-02-10T09:20:26.356 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction test 2024-02-10T09:20:26.406 INFO:teuthology.orchestra.run.smithi158.stdout:Transaction test succeeded. 2024-02-10T09:20:26.409 INFO:teuthology.orchestra.run.smithi158.stdout:Running transaction 2024-02-10T09:20:26.523 INFO:teuthology.orchestra.run.smithi158.stdout: Preparing : 1/1 2024-02-10T09:20:26.701 INFO:teuthology.orchestra.run.smithi158.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:26.996 INFO:teuthology.orchestra.run.smithi158.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:27.262 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T09:20:27.309 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:27.310 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-02-10T09:20:27.310 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:27.396 INFO:teuthology.orchestra.run.smithi158.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:27.397 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:27.397 INFO:teuthology.orchestra.run.smithi158.stdout:Installed: 2024-02-10T09:20:27.397 INFO:teuthology.orchestra.run.smithi158.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:27.397 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:20:27.397 INFO:teuthology.orchestra.run.smithi158.stdout:Complete! 2024-02-10T09:20:27.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-02-10T09:20:27.625 DEBUG:teuthology.parallel:result is None 2024-02-10T09:20:28.188 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:30 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:29.703 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:20:29.704 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:29.705 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:20:29.705 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:29.705 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 89 k 2024-02-10T09:20:29.705 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 298 k 2024-02-10T09:20:29.705 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:20:29.752 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-02-10T09:20:29.753 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:29.753 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.8 MB/s | 89 kB 00:00 2024-02-10T09:20:29.753 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:20:29.762 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:20:29.762 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:20:29.812 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:20:29.815 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:20:29.920 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:20:30.120 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:30.408 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:30.792 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:20:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:30.792 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:30.792 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:31.014 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-02-10T09:20:31.772 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:33 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:33.289 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:33.290 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:33.290 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:33.290 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.7 M 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 18 M 2024-02-10T09:20:33.291 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:20:33.380 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-02-10T09:20:33.381 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:33.381 INFO:teuthology.orchestra.run.smithi083.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-02-10T09:20:33.381 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:20:33.394 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:20:33.394 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:20:33.457 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:20:33.460 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:20:33.919 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:20:34.113 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:34.529 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:34.892 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:34.892 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:34.892 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:20:34.892 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:34.893 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:34.893 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:35.172 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-02-10T09:20:35.934 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:37 ago on Sat 10 Feb 2024 09:17:58 AM UTC. 2024-02-10T09:20:37.442 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:37.443 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-02-10T09:20:37.444 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-02-10T09:20:37.444 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-02-10T09:20:37.444 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:37.444 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 173 k 2024-02-10T09:20:37.444 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 649 k 2024-02-10T09:20:37.444 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-02-10T09:20:37.492 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 173 kB 00:00 2024-02-10T09:20:37.493 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-02-10T09:20:37.493 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.4 MB/s | 173 kB 00:00 2024-02-10T09:20:37.493 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-02-10T09:20:37.503 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-02-10T09:20:37.503 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-02-10T09:20:37.552 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-02-10T09:20:37.556 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-02-10T09:20:37.678 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-02-10T09:20:37.869 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:38.163 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:38.599 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T09:20:38.599 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:38.599 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-02-10T09:20:38.599 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-10T09:20:38.599 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-02-10T09:20:38.599 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-02-10T09:20:38.834 DEBUG:teuthology.parallel:result is None 2024-02-10T09:20:38.834 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T09:20:38.835 INFO:teuthology.packaging:ref: None 2024-02-10T09:20:38.835 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T09:20:38.835 INFO:teuthology.packaging:branch: None 2024-02-10T09:20:38.835 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:20:38.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T09:20:38.989 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T09:20:39.011 INFO:teuthology.orchestra.run.smithi083.stdout:16.2.4-0.el8 2024-02-10T09:20:39.012 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-10T09:20:39.012 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-10T09:20:39.014 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T09:20:39.014 INFO:teuthology.packaging:ref: None 2024-02-10T09:20:39.014 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T09:20:39.014 INFO:teuthology.packaging:branch: None 2024-02-10T09:20:39.014 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T09:20:39.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T09:20:39.167 DEBUG:teuthology.orchestra.run.smithi158:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T09:20:39.195 INFO:teuthology.orchestra.run.smithi158.stdout:16.2.4-0.el8 2024-02-10T09:20:39.195 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-10T09:20:39.195 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-10T09:20:39.197 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T09:20:39.197 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:20:39.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T09:20:39.225 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:20:39.225 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T09:20:39.268 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T09:20:39.269 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:20:39.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T09:20:39.296 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T09:20:39.363 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:20:39.363 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T09:20:39.394 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T09:20:39.463 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T09:20:39.463 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:20:39.463 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T09:20:39.491 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T09:20:39.558 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:20:39.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T09:20:39.587 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T09:20:39.656 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T09:20:39.656 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:20:39.656 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T09:20:39.682 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T09:20:39.749 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:20:39.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T09:20:39.778 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T09:20:39.846 INFO:teuthology.run_tasks:Running task print... 2024-02-10T09:20:39.857 INFO:teuthology.task.print:**** done install task... 2024-02-10T09:20:39.857 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T09:20:39.924 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-10T09:20:39.925 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-10T09:20:39.925 INFO:tasks.cephadm:Cluster fsid is a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:20:39.925 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T09:20:39.925 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-10T09:20:39.925 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi083': '172.21.15.83', 'mon.smithi158': '172.21.15.158'} 2024-02-10T09:20:39.925 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T09:20:39.926 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-02-10T09:20:39.952 DEBUG:teuthology.orchestra.run.smithi158:> sudo hostname $(hostname -s) 2024-02-10T09:20:39.980 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-10T09:20:39.981 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T09:20:40.190 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 10 09:20 /home/ubuntu/cephtest/cephadm 2024-02-10T09:20:40.190 DEBUG:teuthology.orchestra.run.smithi158:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T09:20:40.386 INFO:teuthology.orchestra.run.smithi158.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 10 09:20 /home/ubuntu/cephtest/cephadm 2024-02-10T09:20:40.387 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T09:20:40.405 DEBUG:teuthology.orchestra.run.smithi158:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T09:20:40.435 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-02-10T09:20:40.472 DEBUG:teuthology.orchestra.run.smithi158:> sudo mkdir -p /etc/ceph 2024-02-10T09:20:40.499 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-02-10T09:20:40.538 DEBUG:teuthology.orchestra.run.smithi158:> sudo chmod 777 /etc/ceph 2024-02-10T09:20:40.567 INFO:tasks.cephadm:Writing seed config... 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-10T09:20:40.568 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-10T09:20:40.569 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-10T09:20:40.570 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-10T09:20:40.571 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:20:40.571 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T09:20:40.596 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a847c16e-c7f5-11ee-95b8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-10T09:20:40.596 DEBUG:teuthology.orchestra.run.smithi083:mon.smithi083> sudo journalctl -f -n 0 -u ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service 2024-02-10T09:20:40.638 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T09:20:40.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.83 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:20:40.664 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:-- Logs begin at Sat 2024-02-10 09:05:54 UTC. -- 2024-02-10T09:20:40.898 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-10T09:20:41.843 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.8.3 2024-02-10T09:20:41.850 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying podman|docker is present... 2024-02-10T09:20:41.850 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-10T09:20:41.873 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.8.3 2024-02-10T09:20:41.891 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying lvm2 is present... 2024-02-10T09:20:41.891 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying time synchronization is in place... 2024-02-10T09:20:41.891 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled chrony.service 2024-02-10T09:20:41.898 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T09:20:41.899 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active chrony.service 2024-02-10T09:20:41.905 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-02-10T09:20:41.905 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-10T09:20:41.911 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: enabled 2024-02-10T09:20:41.912 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active chronyd.service 2024-02-10T09:20:41.918 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: active 2024-02-10T09:20:41.918 INFO:teuthology.orchestra.run.smithi083.stderr:Unit chronyd.service is enabled and running 2024-02-10T09:20:41.918 INFO:teuthology.orchestra.run.smithi083.stderr:Repeating the final host check... 2024-02-10T09:20:41.918 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-10T09:20:41.939 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 4.8.3 2024-02-10T09:20:41.951 INFO:teuthology.orchestra.run.smithi083.stderr:podman|docker (/bin/podman) is present 2024-02-10T09:20:41.951 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl is present 2024-02-10T09:20:41.951 INFO:teuthology.orchestra.run.smithi083.stderr:lvcreate is present 2024-02-10T09:20:41.951 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled chrony.service 2024-02-10T09:20:41.957 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T09:20:41.957 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active chrony.service 2024-02-10T09:20:41.963 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-02-10T09:20:41.963 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-10T09:20:41.969 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: enabled 2024-02-10T09:20:41.969 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active chronyd.service 2024-02-10T09:20:41.975 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: active 2024-02-10T09:20:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:Unit chronyd.service is enabled and running 2024-02-10T09:20:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:Host looks OK 2024-02-10T09:20:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:Cluster fsid: a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:20:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:Acquiring lock 139768093669864 on /run/cephadm/a847c16e-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:20:41.976 INFO:teuthology.orchestra.run.smithi083.stderr:Lock 139768093669864 acquired on /run/cephadm/a847c16e-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:20:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying IP 172.21.15.83 port 3300 ... 2024-02-10T09:20:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying IP 172.21.15.83 port 6789 ... 2024-02-10T09:20:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:Base mon IP is 172.21.15.83, final addrv is [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-02-10T09:20:41.977 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /sbin/ip route ls 2024-02-10T09:20:41.980 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.83 metric 100 2024-02-10T09:20:41.980 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.83 metric 100 2024-02-10T09:20:41.980 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /sbin/ip -6 route ls 2024-02-10T09:20:41.982 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T09:20:41.983 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T09:20:41.983 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-02-10T09:20:41.983 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /sbin/ip -6 addr ls 2024-02-10T09:20:41.985 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T09:20:41.985 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T09:20:41.985 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T09:20:41.986 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T09:20:41.986 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:ce91/64 scope link 2024-02-10T09:20:41.986 INFO:teuthology.orchestra.run.smithi083.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T09:20:41.986 INFO:teuthology.orchestra.run.smithi083.stderr:Mon IP 172.21.15.83 is in CIDR network 172.21.0.0/20 2024-02-10T09:20:41.986 INFO:teuthology.orchestra.run.smithi083.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T09:20:41.988 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-10T09:20:41.988 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-10T09:20:42.077 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-10T09:20:48.665 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Getting image source signatures 2024-02-10T09:20:48.674 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-10T09:20:48.674 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-10T09:21:00.334 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-10T09:21:00.491 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T09:21:00.796 INFO:teuthology.orchestra.run.smithi083.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-10T09:21:00.808 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-10T09:21:01.433 INFO:teuthology.orchestra.run.smithi083.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-10T09:21:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-10T09:21:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T09:21:01.997 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-10T09:21:02.489 INFO:teuthology.orchestra.run.smithi083.stderr:stat: 167 167 2024-02-10T09:21:03.042 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial keys... 2024-02-10T09:21:03.043 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-10T09:21:03.587 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQD/P8dlrcnBGhAA8fjvMyurfKWQITJYYo18Xw== 2024-02-10T09:21:04.165 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-10T09:21:04.701 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQAAQMdlQSkIJBAAxUAwWH1c0K3X7QOfBtTu4Q== 2024-02-10T09:21:05.332 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-10T09:21:05.832 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-authtool: AQABQMdlZdGRKhAAUlPyjG5tSPyr7nEL31+X/A== 2024-02-10T09:21:06.307 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial monmap... 2024-02-10T09:21:06.307 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp4gc_pffd:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 --addv smithi083 [v2:172.21.15.83:3300,v1:172.21.15.83:6789] /tmp/monmap 2024-02-10T09:21:06.805 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T09:21:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:06.806 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T09:21:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:monmaptool for smithi083 [v2:172.21.15.83:3300,v1:172.21.15.83:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T09:21:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: set fsid to a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:07.391 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T09:21:07.392 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:21:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:Creating mon... 2024-02-10T09:21:07.392 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/mon.smithi083:/var/lib/ceph/mon/ceph-smithi083:z -v /tmp/ceph-tmp8esrdu94:/tmp/keyring:z -v /tmp/ceph-tmp4gc_pffd:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi083 --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-10T09:21:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.826+0000 7f3c6e4c8700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:21:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.826+0000 7f3c6e4c8700 1 imported monmap: 2024-02-10T09:21:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-10T09:21:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: fsid a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:07.900 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: last_changed 2024-02-10T09:21:06.712710+0000 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: created 2024-02-10T09:21:06.712710+0000 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.smithi083 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.826+0000 7f3c6e4c8700 0 /usr/bin/ceph-mon: set fsid to a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Compile date May 13 2021 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: DB SUMMARY 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi083/store.db dir, Total Num: 0, files: 2024-02-10T09:21:07.901 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi083/store.db: 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.create_if_missing: 1 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.env: 0x562a0d5a11c0 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.fs: Posix File System 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.info_log: 0x562a0fdc0ec0 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:21:07.902 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.db_log_dir: 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi083/store.db 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:21:07.903 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.write_buffer_manager: 0x562a0fdc7530 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:21:07.904 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.unordered_write: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.row_cache: None 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.wal_filter: None 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:21:07.905 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:21:07.906 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Compression algorithms supported: 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kZSTD supported: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:21:07.907 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.827+0000 7f3c6e4c8700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000001 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.merge_operator: 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:21:07.908 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a0fcd9cb8) 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_cache: 0x562a0fd0ed10 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-10T09:21:07.909 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-10T09:21:07.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.num_levels: 7 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:21:07.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:21:07.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:21:07.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:21:07.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:21:07.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.ttl: 2592000 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.828+0000 7f3c6e4c8700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.829+0000 7f3c6e4c8700 4 rocksdb: DB pointer 0x562a0fdd3800 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.829+0000 7f3c57361700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.829+0000 7f3c57361700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:21:07.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:21:07.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:07.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:21:07.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:21:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: 2024-02-10T09:21:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.830+0000 7f3c6e4c8700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-10T09:21:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.830+0000 7f3c6e4c8700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-10T09:21:07.920 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph-mon: debug 2024-02-10T09:21:07.830+0000 7f3c6e4c8700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi083 for mon.smithi083 2024-02-10T09:21:08.448 INFO:teuthology.orchestra.run.smithi083.stderr:create mon.smithi083 on 2024-02-10T09:21:08.449 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:08.454 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl disable ceph.target 2024-02-10T09:21:08.461 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-10T09:21:08.548 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl enable ceph.target 2024-02-10T09:21:08.554 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T09:21:08.637 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl start ceph.target 2024-02-10T09:21:08.644 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl enable ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target 2024-02-10T09:21:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target. 2024-02-10T09:21:08.650 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target. 2024-02-10T09:21:08.729 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target 2024-02-10T09:21:08.736 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl daemon-reload 2024-02-10T09:21:08.820 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083 2024-02-10T09:21:08.850 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl reset-failed ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083 2024-02-10T09:21:08.855 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to reset failed state of unit ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service: Unit ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service not loaded. 2024-02-10T09:21:08.856 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl enable ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083 2024-02-10T09:21:08.862 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target.wants/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service → /etc/systemd/system/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@.service. 2024-02-10T09:21:08.948 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083 2024-02-10T09:21:08.987 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:08 smithi083 systemd[1]: Starting Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:21:09.464 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T09:21:09.472 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-02-10T09:21:09.473 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T09:21:09.480 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-02-10T09:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-02-10T09:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T09:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-10T09:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mon to start... 2024-02-10T09:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mon... 2024-02-10T09:21:09.481 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/mon.smithi083:/var/lib/ceph/mon/ceph-smithi083:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-10T09:21:09.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.419374+0000 mon.smithi083 (mon.0) 0 : [INF] mkfs a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:09.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.403232+0000 mon.smithi083 (mon.0) 1 : cluster [INF] mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-02-10T09:21:09.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:09 smithi083 bash[38965]: fcd7bf2ed157c3cedf0eb5adcb52ca46622524710974ce5ae35cf78a2e675f38 2024-02-10T09:21:09.739 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:09 smithi083 systemd[1]: Started Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:21:10.057 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.053+0000 7f5548584700 1 Processor -- start 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.053+0000 7f5548584700 1 -- start start 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.054+0000 7f5548584700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f5540100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.054+0000 7f5548584700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55400fc8c0 con 0x7f55400703b0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.054+0000 7f5546320700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f5540100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.054+0000 7f5546320700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f5540100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:46996/0 (socket says 172.21.15.83:46996) 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.054+0000 7f5546320700 1 -- 172.21.15.83:0/2259285238 learned_addr learned my addr 172.21.15.83:0/2259285238 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f5546320700 1 -- 172.21.15.83:0/2259285238 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55401013c0 con 0x7f55400703b0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f5546320700 1 --2- 172.21.15.83:0/2259285238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f5540100ed0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f553000b5f0 tx=0x7f5530006ee0).ready entity=mon.0 client_cookie=8f94b61a1a2e5056 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f554531e700 1 -- 172.21.15.83:0/2259285238 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f553000d600 con 0x7f55400703b0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f554531e700 1 -- 172.21.15.83:0/2259285238 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553000d760 con 0x7f55400703b0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f554531e700 1 -- 172.21.15.83:0/2259285238 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f553000da30 con 0x7f55400703b0 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f5548584700 1 -- 172.21.15.83:0/2259285238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 msgr2=0x7f5540100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:10.058 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.055+0000 7f5548584700 1 --2- 172.21.15.83:0/2259285238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f5540100ed0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f553000b5f0 tx=0x7f5530006ee0).stop 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 -- 172.21.15.83:0/2259285238 shutdown_connections 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 --2- 172.21.15.83:0/2259285238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f5540100ed0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 -- 172.21.15.83:0/2259285238 >> 172.21.15.83:0/2259285238 conn(0x7f55400f6530 msgr2=0x7f55400f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 -- 172.21.15.83:0/2259285238 shutdown_connections 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 -- 172.21.15.83:0/2259285238 wait complete. 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 Processor -- start 2024-02-10T09:21:10.059 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.056+0000 7f5548584700 1 -- start start 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5548584700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f554010c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5548584700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554010ca40 con 0x7f55400703b0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5546320700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f554010c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5546320700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f554010c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47006/0 (socket says 172.21.15.83:47006) 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5546320700 1 -- 172.21.15.83:0/3390343722 learned_addr learned my addr 172.21.15.83:0/3390343722 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5546320700 1 -- 172.21.15.83:0/3390343722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553000b200 con 0x7f55400703b0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5546320700 1 --2- 172.21.15.83:0/3390343722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f554010c550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5530006c70 tx=0x7f5530016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5530011b40 con 0x7f55400703b0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5530011ca0 con 0x7f55400703b0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554010cc40 con 0x7f55400703b0 2024-02-10T09:21:10.061 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f553001d430 con 0x7f55400703b0 2024-02-10T09:21:10.062 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.057+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554010d0d0 con 0x7f55400703b0 2024-02-10T09:21:10.062 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.058+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f553000f070 con 0x7f55400703b0 2024-02-10T09:21:10.062 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.058+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5530027460 con 0x7f55400703b0 2024-02-10T09:21:10.062 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.058+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5540045ad0 con 0x7f55400703b0 2024-02-10T09:21:10.062 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.061+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5530016070 con 0x7f55400703b0 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.127+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f554010e5e0 con 0x7f55400703b0 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.127+0000 7f552effd700 1 -- 172.21.15.83:0/3390343722 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f5530036070 con 0x7f55400703b0 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: cluster: 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: id: a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: services: 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi083 (age 0.711408s) 2024-02-10T09:21:10.131 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: data: 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: pgs: 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 msgr2=0x7f554010c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 --2- 172.21.15.83:0/3390343722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f554010c550 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5530006c70 tx=0x7f5530016000).stop 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 shutdown_connections 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 --2- 172.21.15.83:0/3390343722 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55400703b0 0x7f554010c550 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 >> 172.21.15.83:0/3390343722 conn(0x7f55400f6530 msgr2=0x7f55400f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:10.132 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 shutdown_connections 2024-02-10T09:21:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:10.128+0000 7f5548584700 1 -- 172.21.15.83:0/3390343722 wait complete. 2024-02-10T09:21:10.678 INFO:teuthology.orchestra.run.smithi083.stderr:mon is available 2024-02-10T09:21:10.678 INFO:teuthology.orchestra.run.smithi083.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T09:21:10.679 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/mon.smithi083:/var/lib/ceph/mon/ceph-smithi083:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi083/config 2024-02-10T09:21:10.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.417900+0000 mon.smithi083 (mon.0) 2 : cluster [INF] mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-02-10T09:21:10.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.418077+0000 mon.smithi083 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} 2024-02-10T09:21:10.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.421170+0000 mon.smithi083 (mon.0) 4 : cluster [DBG] fsmap 2024-02-10T09:21:10.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.423881+0000 mon.smithi083 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T09:21:10.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: cluster 2024-02-10T09:21:09.424244+0000 mon.smithi083 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T09:21:10.680 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: audit 2024-02-10T09:21:10.129272+0000 mon.smithi083 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.83:0/3390343722' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T09:21:11.196 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.176+0000 7f4966bf6700 1 Processor -- start 2024-02-10T09:21:11.196 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.176+0000 7f4966bf6700 1 -- start start 2024-02-10T09:21:11.196 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.176+0000 7f4966bf6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f4960100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:11.196 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.176+0000 7f4966bf6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49600fc880 con 0x7f49600703f0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f4964992700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f4960100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f4964992700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f4960100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47020/0 (socket says 172.21.15.83:47020) 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f4964992700 1 -- 172.21.15.83:0/2205252185 learned_addr learned my addr 172.21.15.83:0/2205252185 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f4964992700 1 -- 172.21.15.83:0/2205252185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4960101380 con 0x7f49600703f0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f4964992700 1 --2- 172.21.15.83:0/2205252185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f4960100e90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f495400b5f0 tx=0x7f4954006ee0).ready entity=mon.0 client_cookie=14542137ed5198b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f495f7fe700 1 -- 172.21.15.83:0/2205252185 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f495400d600 con 0x7f49600703f0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f495f7fe700 1 -- 172.21.15.83:0/2205252185 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495400d760 con 0x7f49600703f0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.177+0000 7f495f7fe700 1 -- 172.21.15.83:0/2205252185 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f495400da30 con 0x7f49600703f0 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 -- 172.21.15.83:0/2205252185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 msgr2=0x7f4960100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 --2- 172.21.15.83:0/2205252185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f4960100e90 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f495400b5f0 tx=0x7f4954006ee0).stop 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 -- 172.21.15.83:0/2205252185 shutdown_connections 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 --2- 172.21.15.83:0/2205252185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f4960100e90 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 -- 172.21.15.83:0/2205252185 >> 172.21.15.83:0/2205252185 conn(0x7f49600f6530 msgr2=0x7f49600f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:11.197 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 -- 172.21.15.83:0/2205252185 shutdown_connections 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 -- 172.21.15.83:0/2205252185 wait complete. 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 Processor -- start 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.178+0000 7f4966bf6700 1 -- start start 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4966bf6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f496010c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4966bf6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496010ca20 con 0x7f49600703f0 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4964992700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f496010c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4964992700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f496010c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47024/0 (socket says 172.21.15.83:47024) 2024-02-10T09:21:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4964992700 1 -- 172.21.15.83:0/455637617 learned_addr learned my addr 172.21.15.83:0/455637617 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4964992700 1 -- 172.21.15.83:0/455637617 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495400b200 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4964992700 1 --2- 172.21.15.83:0/455637617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f496010c530 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f4954010000 tx=0x7f4954016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4954011c80 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4954011de0 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496010cc20 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f495401d5b0 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496010d0b0 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f495400f070 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.179+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4954027460 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.180+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4960045ad0 con 0x7f49600703f0 2024-02-10T09:21:11.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.182+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4954016070 con 0x7f49600703f0 2024-02-10T09:21:11.247 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.246+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f49601061d0 con 0x7f49600703f0 2024-02-10T09:21:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.249+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f495401d710 con 0x7f49600703f0 2024-02-10T09:21:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.249+0000 7f495d7fa700 1 -- 172.21.15.83:0/455637617 <== mon.0 v2:172.21.15.83:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7f4954031a70 con 0x7f49600703f0 2024-02-10T09:21:11.253 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [global] 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: fsid = a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.83:3300,v1:172.21.15.83:6789] 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [mgr] 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: [osd] 2024-02-10T09:21:11.254 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 msgr2=0x7f496010c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 --2- 172.21.15.83:0/455637617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f496010c530 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f4954010000 tx=0x7f4954016000).stop 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 shutdown_connections 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 --2- 172.21.15.83:0/455637617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49600703f0 0x7f496010c530 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 >> 172.21.15.83:0/455637617 conn(0x7f49600f6530 msgr2=0x7f49600f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 shutdown_connections 2024-02-10T09:21:11.255 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:11.250+0000 7f4966bf6700 1 -- 172.21.15.83:0/455637617 wait complete. 2024-02-10T09:21:11.573 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: audit 2024-02-10T09:21:11.247850+0000 mon.smithi083 (mon.0) 8 : audit [INF] from='client.? 172.21.15.83:0/455637617' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T09:21:11.573 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: audit 2024-02-10T09:21:11.250941+0000 mon.smithi083 (mon.0) 9 : audit [INF] from='client.? 172.21.15.83:0/455637617' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T09:21:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:Generating new minimal ceph.conf... 2024-02-10T09:21:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z -v /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/mon.smithi083:/var/lib/ceph/mon/ceph-smithi083:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi083/config 2024-02-10T09:21:12.354 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f9408ee7700 1 Processor -- start 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f9408ee7700 1 -- start start 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f9408ee7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f9408ee7700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040fef80 con 0x7f94040703f0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f940259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f940259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47036/0 (socket says 172.21.15.83:47036) 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f940259c700 1 -- 172.21.15.83:0/4093298408 learned_addr learned my addr 172.21.15.83:0/4093298408 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f940259c700 1 -- 172.21.15.83:0/4093298408 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94040ff0c0 con 0x7f94040703f0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.349+0000 7f940259c700 1 --2- 172.21.15.83:0/4093298408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f93ec00b5f0 tx=0x7f93ec006ee0).ready entity=mon.0 client_cookie=2048ccf815eecc1a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f940159a700 1 -- 172.21.15.83:0/4093298408 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93ec00d600 con 0x7f94040703f0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f940159a700 1 -- 172.21.15.83:0/4093298408 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f93ec00dbf0 con 0x7f94040703f0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f940159a700 1 -- 172.21.15.83:0/4093298408 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93ec011ce0 con 0x7f94040703f0 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 -- 172.21.15.83:0/4093298408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 msgr2=0x7f94040fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:12.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 --2- 172.21.15.83:0/4093298408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fea90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f93ec00b5f0 tx=0x7f93ec006ee0).stop 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 -- 172.21.15.83:0/4093298408 shutdown_connections 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 --2- 172.21.15.83:0/4093298408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fea90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 -- 172.21.15.83:0/4093298408 >> 172.21.15.83:0/4093298408 conn(0x7f94040f6530 msgr2=0x7f94040f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 -- 172.21.15.83:0/4093298408 shutdown_connections 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.350+0000 7f9408ee7700 1 -- 172.21.15.83:0/4093298408 wait complete. 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f9408ee7700 1 Processor -- start 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f9408ee7700 1 -- start start 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f9408ee7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f9408ee7700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040fa990 con 0x7f94040703f0 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f940259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f940259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:47048/0 (socket says 172.21.15.83:47048) 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f940259c700 1 -- 172.21.15.83:0/1875506350 learned_addr learned my addr 172.21.15.83:0/1875506350 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:12.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f940259c700 1 -- 172.21.15.83:0/1875506350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec00b200 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.351+0000 7f940259c700 1 --2- 172.21.15.83:0/1875506350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f93ec011ed0 tx=0x7f93ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93ec01b570 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f93ec01bb60 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93ec01aad0 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94040fab30 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94040fafc0 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f93ec00f070 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.352+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f93ec025cb0 con 0x7f94040703f0 2024-02-10T09:21:12.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.353+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9404045ad0 con 0x7f94040703f0 2024-02-10T09:21:12.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.356+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f93ec016070 con 0x7f94040703f0 2024-02-10T09:21:12.421 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.420+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9404104b00 con 0x7f94040703f0 2024-02-10T09:21:12.421 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.420+0000 7f93faffd700 1 -- 172.21.15.83:0/1875506350 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f93ec025460 con 0x7f94040703f0 2024-02-10T09:21:12.422 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 msgr2=0x7f94040fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:12.422 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 --2- 172.21.15.83:0/1875506350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fc510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f93ec011ed0 tx=0x7f93ec006ee0).stop 2024-02-10T09:21:12.422 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 shutdown_connections 2024-02-10T09:21:12.422 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 --2- 172.21.15.83:0/1875506350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94040703f0 0x7f94040fc510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:12.422 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 >> 172.21.15.83:0/1875506350 conn(0x7f94040f6530 msgr2=0x7f94040f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:12.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 shutdown_connections 2024-02-10T09:21:12.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:12.421+0000 7f9408ee7700 1 -- 172.21.15.83:0/1875506350 wait complete. 2024-02-10T09:21:12.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:12 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: audit 2024-02-10T09:21:12.421435+0000 mon.smithi083 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.83:0/1875506350' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:21:13.434 INFO:teuthology.orchestra.run.smithi083.stderr:Restarting the monitor... 2024-02-10T09:21:13.434 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl restart ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083 2024-02-10T09:21:13.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:13 smithi083 systemd[1]: Stopping Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:21:13.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: debug 2024-02-10T09:21:13.581+0000 7f830dd42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi083 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T09:21:13.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[38980]: debug 2024-02-10T09:21:13.581+0000 7f830dd42700 -1 mon.smithi083@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T09:21:14.178 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:13 smithi083 podman[39306]: ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083 2024-02-10T09:21:14.179 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 systemd[1]: ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service: Succeeded. 2024-02-10T09:21:14.510 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 systemd[1]: Stopped Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:21:14.510 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 systemd[1]: Starting Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:21:14.698 INFO:teuthology.orchestra.run.smithi083.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-10T09:21:14.699 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.631+0000 7fe1e7b2e700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.631+0000 7fe1e7b2e700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.631+0000 7fe1e7b2e700 0 pidfile_write: ignore empty --pid-file 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 0 load: jerasure load: lrc load: isa 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Compile date May 13 2021 2024-02-10T09:21:14.906 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: DB SUMMARY 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: CURRENT file: CURRENT 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi083/store.db dir, Total Num: 1, files: 000004.sst 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi083/store.db: 000006.log size: 98702 ; 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.error_if_exists: 0 2024-02-10T09:21:14.907 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.create_if_missing: 0 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.env: 0x562a1ef3b1c0 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.fs: Posix File System 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.info_log: 0x562a1fc25960 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.statistics: (nil) 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.use_fsync: 0 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.636+0000 7fe1e7b2e700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T09:21:14.908 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.db_log_dir: 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi083/store.db 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T09:21:14.909 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.write_buffer_manager: 0x562a2089a030 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T09:21:14.910 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.unordered_write: 0 2024-02-10T09:21:14.911 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.row_cache: None 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.wal_filter: None 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.two_write_queues: 0 2024-02-10T09:21:14.913 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.atomic_flush: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T09:21:14.914 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_open_files: -1 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Compression algorithms supported: 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kZSTD supported: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kXpressCompression supported: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T09:21:14.915 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kZlibCompression supported: 1 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000005 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.merge_operator: 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_filter: None 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T09:21:14.916 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562a1fb3dcb0) 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cache_index_and_filter_blocks: 1 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: pin_top_level_index_and_filter: 1 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: index_type: 0 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: data_block_index_type: 0 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: index_shortening: 1 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: hash_index_allow_collision: 1 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: checksum: 1 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: no_block_cache: 0 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_cache: 0x562a1fb72f10 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_cache_name: BinnedLRUCache 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_cache_options: 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: capacity : 536870912 2024-02-10T09:21:14.917 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: num_shard_bits : 4 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: strict_capacity_limit : 0 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: high_pri_pool_ratio: 0.000 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_cache_compressed: (nil) 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: persistent_cache: (nil) 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_size: 4096 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_size_deviation: 10 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_restart_interval: 16 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: index_block_restart_interval: 1 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: metadata_block_size: 4096 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: partition_filters: 0 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: use_delta_encoding: 1 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: whole_key_filtering: 1 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: verify_compression: 0 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: read_amp_bytes_per_bit: 0 2024-02-10T09:21:14.918 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: format_version: 2 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: enable_index_compression: 1 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: block_align: 0 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression: NoCompression 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.num_levels: 7 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T09:21:14.919 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T09:21:14.920 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T09:21:14.921 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T09:21:14.922 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.table_properties_collectors: 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.bloom_locality: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T09:21:14.923 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.ttl: 2592000 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi083/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707556874638561, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.637+0000 7fe1e7b2e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.638+0000 7fe1e7b2e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.638+0000 7fe1e7b2e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707556874639385, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94276, "table_properties": {"data_size": 92364, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85395, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707556874, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.638+0000 7fe1e7b2e700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.639+0000 7fe1e7b2e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707556874639794, "job": 1, "event": "recovery_finished"} 2024-02-10T09:21:14.924 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.639+0000 7fe1e7b2e700 4 rocksdb: DB pointer 0x562a1fc37800 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.640+0000 7fe1ce946700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.640+0000 7fe1ce946700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** DB Stats ** 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** Compaction Stats [default] ** 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:14.925 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** Compaction Stats [default] ** 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Cumulative compaction: 0.00 GB write, 32.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:14.926 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Interval compaction: 0.00 GB write, 32.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** Compaction Stats [default] ** 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.927 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** Compaction Stats [default] ** 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.0 0.00 0.00 1 0.001 0 0 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Cumulative compaction: 0.00 GB write, 32.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.928 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.640+0000 7fe1e7b2e700 0 starting mon.smithi083 rank 0 at public addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] at bind addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi083 fsid a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 1 mon.smithi083@-1(???) e1 preinit fsid a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 0 mon.smithi083@-1(???).mds e1 new map 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 0 mon.smithi083@-1(???).mds e1 print_map 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: e1 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: legacy client fscid: -1 2024-02-10T09:21:14.929 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: No filesystems configured 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 0 mon.smithi083@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 0 mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 0 mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 0 mon.smithi083@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:12.421435+0000 mon.smithi083 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.83:0/1875506350' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.641+0000 7fe1e7b2e700 1 mon.smithi083@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expand map: {default=false} 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta from 'false' to 'false' 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expanded map: {default=false} 2024-02-10T09:21:14.930 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expand map: {default=info} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta from 'info' to 'info' 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expanded map: {default=info} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expand map: {default=daemon} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expand map: {default=debug} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:21:14.642+0000 7fe1e7b2e700 20 expand_channel_meta expanded map: {default=debug} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 bash[39375]: 0282f5187d842d87f5c1470f7f9f5202cb5bdbcead48030bdfd5c35fb1e0c22b 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 systemd[1]: Started Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715. 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:14.657199+0000 mon.smithi083 (mon.0) 1 : cluster [INF] mon.smithi083 is new leader, mons smithi083 in quorum (ranks 0) 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:14.657261+0000 mon.smithi083 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi083=[v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0]} 2024-02-10T09:21:14.931 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:14.657289+0000 mon.smithi083 (mon.0) 3 : cluster [DBG] fsmap 2024-02-10T09:21:14.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:14.657312+0000 mon.smithi083 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T09:21:14.932 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:14.657664+0000 mon.smithi083 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T09:21:15.194 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.193+0000 7efe5e58a700 1 Processor -- start 2024-02-10T09:21:15.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.193+0000 7efe5e58a700 1 -- start start 2024-02-10T09:21:15.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe5e58a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe580fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:15.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe5e58a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe580fb6a0 con 0x7efe580fad90 2024-02-10T09:21:15.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe57fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe580fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:15.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe57fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe580fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41828/0 (socket says 172.21.15.83:41828) 2024-02-10T09:21:15.199 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe57fff700 1 -- 172.21.15.83:0/3497069486 learned_addr learned my addr 172.21.15.83:0/3497069486 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe57fff700 1 -- 172.21.15.83:0/3497069486 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580fb7e0 con 0x7efe580fad90 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.194+0000 7efe57fff700 1 --2- 172.21.15.83:0/3497069486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe580fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7efe4800b5f0 tx=0x7efe48006ee0).ready entity=mon.0 client_cookie=e30b506d6b70c188 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe56ffd700 1 -- 172.21.15.83:0/3497069486 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4800d600 con 0x7efe580fad90 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe56ffd700 1 -- 172.21.15.83:0/3497069486 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efe4800dbf0 con 0x7efe580fad90 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe56ffd700 1 -- 172.21.15.83:0/3497069486 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe48011cb0 con 0x7efe580fad90 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 -- 172.21.15.83:0/3497069486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 msgr2=0x7efe580fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 --2- 172.21.15.83:0/3497069486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe580fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7efe4800b5f0 tx=0x7efe48006ee0).stop 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 -- 172.21.15.83:0/3497069486 shutdown_connections 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 --2- 172.21.15.83:0/3497069486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe580fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 -- 172.21.15.83:0/3497069486 >> 172.21.15.83:0/3497069486 conn(0x7efe580f6530 msgr2=0x7efe580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 -- 172.21.15.83:0/3497069486 shutdown_connections 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.195+0000 7efe5e58a700 1 -- 172.21.15.83:0/3497069486 wait complete. 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.196+0000 7efe5e58a700 1 Processor -- start 2024-02-10T09:21:15.200 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.196+0000 7efe5e58a700 1 -- start start 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.196+0000 7efe5e58a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe58109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.196+0000 7efe5e58a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe58109f80 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.196+0000 7efe57fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe58109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe57fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe58109a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41842/0 (socket says 172.21.15.83:41842) 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe57fff700 1 -- 172.21.15.83:0/3213307434 learned_addr learned my addr 172.21.15.83:0/3213307434 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe57fff700 1 -- 172.21.15.83:0/3213307434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4800b200 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe57fff700 1 --2- 172.21.15.83:0/3213307434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe58109a90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7efe48002f40 tx=0x7efe48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4801b410 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efe4801ba00 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5810a180 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4801a900 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.197+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5810a610 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.198+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7efe4800f070 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.198+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7efe480257d0 con 0x7efe580fad90 2024-02-10T09:21:15.201 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.198+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe58045ad0 con 0x7efe580fad90 2024-02-10T09:21:15.202 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.200+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7efe48016070 con 0x7efe580fad90 2024-02-10T09:21:15.264 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.263+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7efe580fcae0 con 0x7efe580fad90 2024-02-10T09:21:15.266 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.266+0000 7efe54ff9700 1 -- 172.21.15.83:0/3213307434 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7efe4801aaf0 con 0x7efe580fad90 2024-02-10T09:21:15.267 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.266+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 msgr2=0x7efe58109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:15.267 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.266+0000 7efe5e58a700 1 --2- 172.21.15.83:0/3213307434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe58109a90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7efe48002f40 tx=0x7efe48006ee0).stop 2024-02-10T09:21:15.268 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.266+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 shutdown_connections 2024-02-10T09:21:15.268 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.266+0000 7efe5e58a700 1 --2- 172.21.15.83:0/3213307434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efe580fad90 0x7efe58109a90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:15.268 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.266+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 >> 172.21.15.83:0/3213307434 conn(0x7efe580f6530 msgr2=0x7efe580f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:15.268 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.267+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 shutdown_connections 2024-02-10T09:21:15.268 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:15.267+0000 7efe5e58a700 1 -- 172.21.15.83:0/3213307434 wait complete. 2024-02-10T09:21:15.846 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T09:21:15.846 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:21:15.846 INFO:teuthology.orchestra.run.smithi083.stderr:Creating mgr... 2024-02-10T09:21:15.847 INFO:teuthology.orchestra.run.smithi083.stderr:Verifying port 9283 ... 2024-02-10T09:21:15.849 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl daemon-reload 2024-02-10T09:21:15.940 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym 2024-02-10T09:21:15.961 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl reset-failed ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym 2024-02-10T09:21:15.966 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Failed to reset failed state of unit ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym.service: Unit ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym.service not loaded. 2024-02-10T09:21:15.967 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl enable ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym 2024-02-10T09:21:15.973 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: Created symlink /etc/systemd/system/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715.target.wants/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym.service → /etc/systemd/system/ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@.service. 2024-02-10T09:21:16.056 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mgr.smithi083.dowdym 2024-02-10T09:21:16.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:16 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:15.266586+0000 mon.smithi083 (mon.0) 6 : audit [INF] from='client.? 172.21.15.83:0/3213307434' entity='client.admin' 2024-02-10T09:21:16.660 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T09:21:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-02-10T09:21:16.670 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T09:21:16.679 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-02-10T09:21:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-02-10T09:21:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T09:21:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-10T09:21:16.680 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T09:21:16.689 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-02-10T09:21:16.689 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T09:21:16.698 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-02-10T09:21:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-02-10T09:21:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T09:21:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr to start... 2024-02-10T09:21:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr... 2024-02-10T09:21:16.699 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.232+0000 7fe10609c700 1 Processor -- start 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.232+0000 7fe10609c700 1 -- start start 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.232+0000 7fe10609c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe1000feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.232+0000 7fe10609c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1000fefa0 con 0x7fe1000703d0 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0ff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe1000feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0ff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe1000feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41872/0 (socket says 172.21.15.83:41872) 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0ff7fe700 1 -- 172.21.15.83:0/1671945690 learned_addr learned my addr 172.21.15.83:0/1671945690 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0ff7fe700 1 -- 172.21.15.83:0/1671945690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000ff0e0 con 0x7fe1000703d0 2024-02-10T09:21:17.235 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0ff7fe700 1 --2- 172.21.15.83:0/1671945690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe1000feab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe0f000b5f0 tx=0x7fe0f0006ee0).ready entity=mon.0 client_cookie=2b9503d25a08825a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0fe7fc700 1 -- 172.21.15.83:0/1671945690 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f000d600 con 0x7fe1000703d0 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0fe7fc700 1 -- 172.21.15.83:0/1671945690 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe0f000dbf0 con 0x7fe1000703d0 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.233+0000 7fe0fe7fc700 1 -- 172.21.15.83:0/1671945690 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f0011cb0 con 0x7fe1000703d0 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 -- 172.21.15.83:0/1671945690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 msgr2=0x7fe1000feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 --2- 172.21.15.83:0/1671945690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe1000feab0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe0f000b5f0 tx=0x7fe0f0006ee0).stop 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 -- 172.21.15.83:0/1671945690 shutdown_connections 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 --2- 172.21.15.83:0/1671945690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe1000feab0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 -- 172.21.15.83:0/1671945690 >> 172.21.15.83:0/1671945690 conn(0x7fe1000f6530 msgr2=0x7fe1000f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 -- 172.21.15.83:0/1671945690 shutdown_connections 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 -- 172.21.15.83:0/1671945690 wait complete. 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.234+0000 7fe10609c700 1 Processor -- start 2024-02-10T09:21:17.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe10609c700 1 -- start start 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe10609c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe10010ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe10609c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10010b060 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe0ff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe10010ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe0ff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe10010ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41878/0 (socket says 172.21.15.83:41878) 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe0ff7fe700 1 -- 172.21.15.83:0/2639244364 learned_addr learned my addr 172.21.15.83:0/2639244364 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe0ff7fe700 1 -- 172.21.15.83:0/2639244364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f000b200 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.235+0000 7fe0ff7fe700 1 --2- 172.21.15.83:0/2639244364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe10010ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe0f0006a20 tx=0x7fe0f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f001b570 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10010b260 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe0f001bb60 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f001aa20 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10010b6f0 con 0x7fe1000703d0 2024-02-10T09:21:17.239 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe0f000f070 con 0x7fe1000703d0 2024-02-10T09:21:17.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.236+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe0f0025de0 con 0x7fe1000703d0 2024-02-10T09:21:17.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.237+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe100045ad0 con 0x7fe1000703d0 2024-02-10T09:21:17.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.239+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe0f0016070 con 0x7fe1000703d0 2024-02-10T09:21:17.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.308+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe1000fff70 con 0x7fe1000703d0 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.308+0000 7fe0e7fff700 1 -- 172.21.15.83:0/2639244364 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fe0f0025460 con 0x7fe1000703d0 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "a847c16e-c7f5-11ee-95b8-87774f69a715", 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-02-10T09:21:17.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "smithi083" 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T09:21:17.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": false, 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T09:21:17.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-02-10T09:21:09.419698+0000", 2024-02-10T09:21:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-02-10T09:21:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T09:21:17.319 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 msgr2=0x7fe10010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 --2- 172.21.15.83:0/2639244364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe10010ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe0f0006a20 tx=0x7fe0f0006ee0).stop 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 shutdown_connections 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 --2- 172.21.15.83:0/2639244364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1000703d0 0x7fe10010ab70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 >> 172.21.15.83:0/2639244364 conn(0x7fe1000f6530 msgr2=0x7fe1000f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 shutdown_connections 2024-02-10T09:21:17.320 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:17.310+0000 7fe10609c700 1 -- 172.21.15.83:0/2639244364 wait complete. 2024-02-10T09:21:17.670 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:17.309121+0000 mon.smithi083 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.83:0/2639244364' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T09:21:17.947 INFO:teuthology.orchestra.run.smithi083.stderr:mgr not available, waiting (1/15)... 2024-02-10T09:21:19.949 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-10T09:21:20.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.418+0000 7fd51eab2700 1 Processor -- start 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51eab2700 1 -- start start 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51eab2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd5180fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51eab2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5180fef40 con 0x7fd518070430 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51c84e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd5180fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51c84e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd5180fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41888/0 (socket says 172.21.15.83:41888) 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51c84e700 1 -- 172.21.15.83:0/2006962430 learned_addr learned my addr 172.21.15.83:0/2006962430 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51c84e700 1 -- 172.21.15.83:0/2006962430 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5180ff080 con 0x7fd518070430 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.419+0000 7fd51c84e700 1 --2- 172.21.15.83:0/2006962430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd5180fea50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd50c00b5f0 tx=0x7fd50c006ee0).ready entity=mon.0 client_cookie=3b4e644b41fc6075 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd5177fe700 1 -- 172.21.15.83:0/2006962430 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50c00d600 con 0x7fd518070430 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd5177fe700 1 -- 172.21.15.83:0/2006962430 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd50c00dbf0 con 0x7fd518070430 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd5177fe700 1 -- 172.21.15.83:0/2006962430 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50c011cb0 con 0x7fd518070430 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 -- 172.21.15.83:0/2006962430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 msgr2=0x7fd5180fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:20.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 --2- 172.21.15.83:0/2006962430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd5180fea50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd50c00b5f0 tx=0x7fd50c006ee0).stop 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 -- 172.21.15.83:0/2006962430 shutdown_connections 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 --2- 172.21.15.83:0/2006962430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd5180fea50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 -- 172.21.15.83:0/2006962430 >> 172.21.15.83:0/2006962430 conn(0x7fd5180f6530 msgr2=0x7fd5180f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 -- 172.21.15.83:0/2006962430 shutdown_connections 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.420+0000 7fd51eab2700 1 -- 172.21.15.83:0/2006962430 wait complete. 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51eab2700 1 Processor -- start 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51eab2700 1 -- start start 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51eab2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd51810ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51eab2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51810b030 con 0x7fd518070430 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51c84e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd51810ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51c84e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd51810ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:41892/0 (socket says 172.21.15.83:41892) 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.421+0000 7fd51c84e700 1 -- 172.21.15.83:0/4241285171 learned_addr learned my addr 172.21.15.83:0/4241285171 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd51c84e700 1 -- 172.21.15.83:0/4241285171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c00b200 con 0x7fd518070430 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd51c84e700 1 --2- 172.21.15.83:0/4241285171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd51810ab40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd50c006800 tx=0x7fd50c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:20.445 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50c01b570 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd51eab2700 1 -- 172.21.15.83:0/4241285171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51810b230 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd51eab2700 1 -- 172.21.15.83:0/4241285171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51810b6c0 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd50c01bb60 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd50c025970 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd50c00f070 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.422+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd50c01a370 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.423+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f0004fa0 con 0x7fd518070430 2024-02-10T09:21:20.446 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.425+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd50c016070 con 0x7fd518070430 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.493+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd4f0005670 con 0x7fd518070430 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.494+0000 7fd5157fa700 1 -- 172.21.15.83:0/4241285171 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fd50c024eb0 con 0x7fd518070430 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "a847c16e-c7f5-11ee-95b8-87774f69a715", 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-02-10T09:21:20.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "smithi083" 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T09:21:20.502 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T09:21:20.503 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T09:21:20.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": false, 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-02-10T09:21:09.419698+0000", 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-02-10T09:21:20.505 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 msgr2=0x7fd51810ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 --2- 172.21.15.83:0/4241285171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd51810ab40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fd50c006800 tx=0x7fd50c006ee0).stop 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 shutdown_connections 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 --2- 172.21.15.83:0/4241285171 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd518070430 0x7fd51810ab40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 >> 172.21.15.83:0/4241285171 conn(0x7fd5180f6530 msgr2=0x7fd5180f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 shutdown_connections 2024-02-10T09:21:20.506 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:20.495+0000 7fd502ffd700 1 -- 172.21.15.83:0/4241285171 wait complete. 2024-02-10T09:21:20.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:20 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:20.494683+0000 mon.smithi083 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.83:0/4241285171' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T09:21:21.029 INFO:teuthology.orchestra.run.smithi083.stderr:mgr not available, waiting (2/15)... 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:21.478161+0000 mon.smithi083 (mon.0) 9 : cluster [INF] Activating manager daemon smithi083.dowdym 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:21.480775+0000 mon.smithi083 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi083.dowdym(active, starting, since 0.00269735s) 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.481720+0000 mon.smithi083 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.481908+0000 mon.smithi083 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.482076+0000 mon.smithi083 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.482244+0000 mon.smithi083 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.482427+0000 mon.smithi083 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mgr metadata", "who": "smithi083.dowdym", "id": "smithi083.dowdym"}]: dispatch 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:21.497246+0000 mon.smithi083 (mon.0) 16 : cluster [INF] Manager daemon smithi083.dowdym is now available 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.505233+0000 mon.smithi083 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.508668+0000 mon.smithi083 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:21:22.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.509707+0000 mon.smithi083 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:21:22.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.513558+0000 mon.smithi083 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:22.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:21.515889+0000 mon.smithi083 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.83:0/1245410089' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:23.032 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-10T09:21:23.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:22.483954+0000 mon.smithi083 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi083.dowdym(active, since 1.00587s) 2024-02-10T09:21:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.531+0000 7f80c7178700 1 Processor -- start 2024-02-10T09:21:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.531+0000 7f80c7178700 1 -- start start 2024-02-10T09:21:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.531+0000 7f80c7178700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.531+0000 7f80c7178700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00fc8c0 con 0x7f80c00703b0 2024-02-10T09:21:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80c4f14700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:23.910 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80c4f14700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60362/0 (socket says 172.21.15.83:60362) 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80c4f14700 1 -- 172.21.15.83:0/267490296 learned_addr learned my addr 172.21.15.83:0/267490296 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80c4f14700 1 -- 172.21.15.83:0/267490296 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c01013c0 con 0x7f80c00703b0 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80c4f14700 1 --2- 172.21.15.83:0/267490296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c0100ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f80b400b5f0 tx=0x7f80b4006ee0).ready entity=mon.0 client_cookie=856d85569ce622e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80bf7fe700 1 -- 172.21.15.83:0/267490296 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b400d600 con 0x7f80c00703b0 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80bf7fe700 1 -- 172.21.15.83:0/267490296 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f80b400dbf0 con 0x7f80c00703b0 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.532+0000 7f80bf7fe700 1 -- 172.21.15.83:0/267490296 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b4011cb0 con 0x7f80c00703b0 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 -- 172.21.15.83:0/267490296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 msgr2=0x7f80c0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 --2- 172.21.15.83:0/267490296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c0100ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f80b400b5f0 tx=0x7f80b4006ee0).stop 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 -- 172.21.15.83:0/267490296 shutdown_connections 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 --2- 172.21.15.83:0/267490296 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c0100ed0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 -- 172.21.15.83:0/267490296 >> 172.21.15.83:0/267490296 conn(0x7f80c00f6530 msgr2=0x7f80c00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 -- 172.21.15.83:0/267490296 shutdown_connections 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 -- 172.21.15.83:0/267490296 wait complete. 2024-02-10T09:21:23.911 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 Processor -- start 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.533+0000 7f80c7178700 1 -- start start 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c7178700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c7178700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c010d270 con 0x7f80c00703b0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c4f14700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c4f14700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60374/0 (socket says 172.21.15.83:60374) 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c4f14700 1 -- 172.21.15.83:0/2661974100 learned_addr learned my addr 172.21.15.83:0/2661974100 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c4f14700 1 -- 172.21.15.83:0/2661974100 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b400b200 con 0x7f80c00703b0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c4f14700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c010cd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f80b4006800 tx=0x7f80b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b401b570 con 0x7f80c00703b0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f80b401bb60 con 0x7f80c00703b0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c010d470 con 0x7f80c00703b0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80b401a9f0 con 0x7f80c00703b0 2024-02-10T09:21:23.912 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.534+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c010e080 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.535+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f80b400f070 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.535+0000 7f80bd7fa700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f80a402fd60 0x7f80a4032220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.535+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f80b40411a0 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.535+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c0059100 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.535+0000 7f80bffff700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f80a402fd60 0x7f80a4032220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.536+0000 7f80bffff700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f80a402fd60 0x7f80a4032220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f80ac0096a0 tx=0x7f80ac006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.540+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80b4016070 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.743+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f80c0045ad0 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.743+0000 7f80bd7fa700 1 -- 172.21.15.83:0/2661974100 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f80b4025460 con 0x7f80c00703b0 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f80a402fd60 msgr2=0x7f80a4032220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f80a402fd60 0x7f80a4032220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f80ac0096a0 tx=0x7f80ac006b40).stop 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 msgr2=0x7f80c010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:23.913 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c010cd80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f80b4006800 tx=0x7f80b4006ee0).stop 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 shutdown_connections 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f80a402fd60 0x7f80a4032220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 --2- 172.21.15.83:0/2661974100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f80c00703b0 0x7f80c010cd80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 >> 172.21.15.83:0/2661974100 conn(0x7f80c00f6530 msgr2=0x7f80c00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 shutdown_connections 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.745+0000 7f80c7178700 1 -- 172.21.15.83:0/2661974100 wait complete. 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsid": "a847c16e-c7f5-11ee-95b8-87774f69a715", 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "health": { 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T09:21:23.914 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 0 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "smithi083" 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-02-10T09:21:23.915 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T09:21:23.916 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T09:21:23.917 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "iostat", 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "restful" 2024-02-10T09:21:23.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ], 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "modified": "2024-02-10T09:21:09.419698+0000", 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "services": {} 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: }, 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T09:21:23.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:21:24.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:23.744383+0000 mon.smithi083 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.83:0/2661974100' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T09:21:25.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:24.484093+0000 mon.smithi083 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi083.dowdym(active, since 3s) 2024-02-10T09:21:28.677 INFO:teuthology.orchestra.run.smithi083.stderr:mgr is available 2024-02-10T09:21:28.678 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling cephadm module... 2024-02-10T09:21:28.678 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.209+0000 7f9fcd234700 1 Processor -- start 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.209+0000 7f9fcd234700 1 -- start start 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.209+0000 7f9fcd234700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc80fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.209+0000 7f9fcd234700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc80fc7a0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc6d9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc80fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc6d9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc80fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60386/0 (socket says 172.21.15.83:60386) 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc6d9d700 1 -- 172.21.15.83:0/1479638568 learned_addr learned my addr 172.21.15.83:0/1479638568 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc6d9d700 1 -- 172.21.15.83:0/1479638568 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc80fc8e0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc6d9d700 1 --2- 172.21.15.83:0/1479638568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc80fc2b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f9fbc00b5f0 tx=0x7f9fbc006ee0).ready entity=mon.0 client_cookie=6290628fe02d9953 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc5d9b700 1 -- 172.21.15.83:0/1479638568 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fbc00d600 con 0x7f9fc80fbe70 2024-02-10T09:21:30.214 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc5d9b700 1 -- 172.21.15.83:0/1479638568 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9fbc00dbf0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.210+0000 7f9fc5d9b700 1 -- 172.21.15.83:0/1479638568 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fbc011cb0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 -- 172.21.15.83:0/1479638568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 msgr2=0x7f9fc80fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 --2- 172.21.15.83:0/1479638568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc80fc2b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f9fbc00b5f0 tx=0x7f9fbc006ee0).stop 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 -- 172.21.15.83:0/1479638568 shutdown_connections 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 --2- 172.21.15.83:0/1479638568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc80fc2b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 -- 172.21.15.83:0/1479638568 >> 172.21.15.83:0/1479638568 conn(0x7f9fc80f7660 msgr2=0x7f9fc80f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 -- 172.21.15.83:0/1479638568 shutdown_connections 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.211+0000 7f9fcd234700 1 -- 172.21.15.83:0/1479638568 wait complete. 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fcd234700 1 Processor -- start 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fcd234700 1 -- start start 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fcd234700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc810ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fcd234700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc810b020 con 0x7f9fc80fbe70 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fc6d9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc810ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:30.215 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fc6d9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc810ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60390/0 (socket says 172.21.15.83:60390) 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fc6d9d700 1 -- 172.21.15.83:0/751156205 learned_addr learned my addr 172.21.15.83:0/751156205 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.212+0000 7f9fc6d9d700 1 -- 172.21.15.83:0/751156205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc00b200 con 0x7f9fc80fbe70 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.213+0000 7f9fc6d9d700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc810ab30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f9fbc006800 tx=0x7f9fbc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.213+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fbc01b570 con 0x7f9fc80fbe70 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.213+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc810b220 con 0x7f9fc80fbe70 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.213+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9fbc01bb60 con 0x7f9fc80fbe70 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.213+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fbc01a9f0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.216 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.213+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc810b6b0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.217 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.214+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f9fbc00f070 con 0x7f9fc80fbe70 2024-02-10T09:21:30.217 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.214+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8045ad0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.217 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.214+0000 7f9faf7fe700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f9fb402fdc0 0x7f9fb4032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:30.217 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.214+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9fbc041410 con 0x7f9fc80fbe70 2024-02-10T09:21:30.217 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.214+0000 7f9fc659c700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f9fb402fdc0 0x7f9fb4032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:30.217 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.214+0000 7f9fc659c700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f9fb402fdc0 0x7f9fb4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9fb80096a0 tx=0x7f9fb8006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:30.219 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.218+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fbc019cd0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.405 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.404+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9fc80fdbe0 con 0x7f9fc80fbe70 2024-02-10T09:21:30.651 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.650+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9fbc01ab50 con 0x7f9fc80fbe70 2024-02-10T09:21:30.652 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.651+0000 7f9faf7fe700 1 -- 172.21.15.83:0/751156205 <== mon.0 v2:172.21.15.83:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9fbc024d90 con 0x7f9fc80fbe70 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f9fb402fdc0 msgr2=0x7f9fb4032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f9fb402fdc0 0x7f9fb4032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9fb80096a0 tx=0x7f9fb8006b40).stop 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 msgr2=0x7f9fc810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc810ab30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f9fbc006800 tx=0x7f9fbc006ee0).stop 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 shutdown_connections 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f9fb402fdc0 0x7f9fb4032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 --2- 172.21.15.83:0/751156205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fc80fbe70 0x7f9fc810ab30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 >> 172.21.15.83:0/751156205 conn(0x7f9fc80f7660 msgr2=0x7f9fc80f8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 shutdown_connections 2024-02-10T09:21:30.655 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:30.653+0000 7f9fcd234700 1 -- 172.21.15.83:0/751156205 wait complete. 2024-02-10T09:21:30.997 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:30 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:30.405613+0000 mon.smithi083 (mon.0) 25 : audit [INF] from='client.? 172.21.15.83:0/751156205' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T09:21:31.275 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-10T09:21:31.792 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d81755700 1 Processor -- start 2024-02-10T09:21:31.792 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d81755700 1 -- start start 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d81755700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d81755700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0fddd0 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60408/0 (socket says 172.21.15.83:60408) 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.788+0000 7f3d7affd700 1 -- 172.21.15.83:0/3766439625 learned_addr learned my addr 172.21.15.83:0/3766439625 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.789+0000 7f3d7affd700 1 -- 172.21.15.83:0/3766439625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c0fdf10 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.789+0000 7f3d7affd700 1 --2- 172.21.15.83:0/3766439625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fd8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3d6c00b5f0 tx=0x7f3d6c006ee0).ready entity=mon.0 client_cookie=758899355fe46ac5 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.789+0000 7f3d79ffb700 1 -- 172.21.15.83:0/3766439625 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c00d600 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.789+0000 7f3d79ffb700 1 -- 172.21.15.83:0/3766439625 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3d6c00dbf0 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.789+0000 7f3d79ffb700 1 -- 172.21.15.83:0/3766439625 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c011cb0 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 -- 172.21.15.83:0/3766439625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 msgr2=0x7f3d7c0fd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 --2- 172.21.15.83:0/3766439625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fd8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3d6c00b5f0 tx=0x7f3d6c006ee0).stop 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 -- 172.21.15.83:0/3766439625 shutdown_connections 2024-02-10T09:21:31.793 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 --2- 172.21.15.83:0/3766439625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fd8e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 -- 172.21.15.83:0/3766439625 >> 172.21.15.83:0/3766439625 conn(0x7f3d7c0f5d20 msgr2=0x7f3d7c0f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 -- 172.21.15.83:0/3766439625 shutdown_connections 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 -- 172.21.15.83:0/3766439625 wait complete. 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 Processor -- start 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.790+0000 7f3d81755700 1 -- start start 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d81755700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d81755700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0fb210 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d7affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fcd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60416/0 (socket says 172.21.15.83:60416) 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d7affd700 1 -- 172.21.15.83:0/2334374589 learned_addr learned my addr 172.21.15.83:0/2334374589 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d7affd700 1 -- 172.21.15.83:0/2334374589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c00b200 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d7affd700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fcd90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3d6c002f40 tx=0x7f3d6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c01b410 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c0fb3b0 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.794 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3d6c01ba00 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c01a980 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.791+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c0fb840 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.792+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3d6c00f070 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.796 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.792+0000 7f3d637fe700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 0x7f3d680322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:31.796 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.792+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3d6c0421b0 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.796 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.792+0000 7f3d7a7fc700 1 -- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 msgr2=0x7f3d680322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/3127942540 2024-02-10T09:21:31.796 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.792+0000 7f3d7a7fc700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 0x7f3d680322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T09:21:31.796 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.793+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c059100 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.797 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.797+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d6c016070 con 0x7f3d7c0fd4c0 2024-02-10T09:21:31.994 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.993+0000 7f3d7a7fc700 1 -- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 msgr2=0x7f3d680322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/3127942540 2024-02-10T09:21:31.994 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:31.993+0000 7f3d7a7fc700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 0x7f3d680322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T09:21:32.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:30.652271+0000 mon.smithi083 (mon.0) 26 : audit [INF] from='client.? 172.21.15.83:0/751156205' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T09:21:32.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:30.652321+0000 mon.smithi083 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi083.dowdym(active, since 9s) 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.038+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3d7c104600 con 0x7f3d7c0fd4c0 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.038+0000 7f3d637fe700 1 -- 172.21.15.83:0/2334374589 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3d6c025460 con 0x7f3d7c0fd4c0 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "active_name": "smithi083.dowdym", 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T09:21:32.039 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:21:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 msgr2=0x7f3d680322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T09:21:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 0x7f3d680322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:21:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 msgr2=0x7f3d7c0fcd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fcd90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3d6c002f40 tx=0x7f3d6c006ee0).stop 2024-02-10T09:21:32.041 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 shutdown_connections 2024-02-10T09:21:32.042 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7f3d6802fe10 0x7f3d680322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:21:32.042 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 --2- 172.21.15.83:0/2334374589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d7c0fd4c0 0x7f3d7c0fcd90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:32.042 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 >> 172.21.15.83:0/2334374589 conn(0x7f3d7c0f5d20 msgr2=0x7f3d7c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:32.042 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 shutdown_connections 2024-02-10T09:21:32.042 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:32.040+0000 7f3d81755700 1 -- 172.21.15.83:0/2334374589 wait complete. 2024-02-10T09:21:32.616 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for the mgr to restart... 2024-02-10T09:21:32.616 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr epoch 5... 2024-02-10T09:21:32.616 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-10T09:21:33.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:32 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:32.038909+0000 mon.smithi083 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.83:0/2334374589' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa33a14700 1 Processor -- start 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa33a14700 1 -- start start 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa33a14700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa33a14700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0fef40 con 0x7ffa2c070430 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa317b0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa317b0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60422/0 (socket says 172.21.15.83:60422) 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.183+0000 7ffa317b0700 1 -- 172.21.15.83:0/397624347 learned_addr learned my addr 172.21.15.83:0/397624347 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.184+0000 7ffa317b0700 1 -- 172.21.15.83:0/397624347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0ff080 con 0x7ffa2c070430 2024-02-10T09:21:33.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.184+0000 7ffa317b0700 1 --2- 172.21.15.83:0/397624347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c0fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ffa2800b5f0 tx=0x7ffa28006ee0).ready entity=mon.0 client_cookie=de89e98e6664e373 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.184+0000 7ffa1bfff700 1 -- 172.21.15.83:0/397624347 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2800d600 con 0x7ffa2c070430 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.184+0000 7ffa1bfff700 1 -- 172.21.15.83:0/397624347 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ffa2800dbf0 con 0x7ffa2c070430 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.184+0000 7ffa1bfff700 1 -- 172.21.15.83:0/397624347 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa28011cb0 con 0x7ffa2c070430 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 -- 172.21.15.83:0/397624347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 msgr2=0x7ffa2c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 --2- 172.21.15.83:0/397624347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c0fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ffa2800b5f0 tx=0x7ffa28006ee0).stop 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 -- 172.21.15.83:0/397624347 shutdown_connections 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 --2- 172.21.15.83:0/397624347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c0fea50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 -- 172.21.15.83:0/397624347 >> 172.21.15.83:0/397624347 conn(0x7ffa2c0f6530 msgr2=0x7ffa2c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 -- 172.21.15.83:0/397624347 shutdown_connections 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.185+0000 7ffa33a14700 1 -- 172.21.15.83:0/397624347 wait complete. 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa33a14700 1 Processor -- start 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa33a14700 1 -- start start 2024-02-10T09:21:33.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa33a14700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:33.189 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa33a14700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c10b030 con 0x7ffa2c070430 2024-02-10T09:21:33.189 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa317b0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:33.189 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa317b0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c10ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60436/0 (socket says 172.21.15.83:60436) 2024-02-10T09:21:33.189 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa317b0700 1 -- 172.21.15.83:0/679279261 learned_addr learned my addr 172.21.15.83:0/679279261 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:33.189 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa317b0700 1 -- 172.21.15.83:0/679279261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2800b200 con 0x7ffa2c070430 2024-02-10T09:21:33.189 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa317b0700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c10ab40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ffa28006800 tx=0x7ffa28006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.186+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2801b570 con 0x7ffa2c070430 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.187+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c10b230 con 0x7ffa2c070430 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.187+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ffa2801bb60 con 0x7ffa2c070430 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.187+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2801a9f0 con 0x7ffa2c070430 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.187+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c10b6c0 con 0x7ffa2c070430 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.187+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ffa2800f070 con 0x7ffa2c070430 2024-02-10T09:21:33.190 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.188+0000 7ffa19ffb700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:33.191 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.188+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 --> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa2c059100 con 0x7ffa1402fe70 2024-02-10T09:21:33.191 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.188+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ffa280413f0 con 0x7ffa2c070430 2024-02-10T09:21:33.191 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.188+0000 7ffa30faf700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 msgr2=0x7ffa14032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/3127942540 2024-02-10T09:21:33.191 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.188+0000 7ffa30faf700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T09:21:33.389 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.389+0000 7ffa30faf700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 msgr2=0x7ffa14032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/3127942540 2024-02-10T09:21:33.390 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.389+0000 7ffa30faf700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T09:21:33.790 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.790+0000 7ffa30faf700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 msgr2=0x7ffa14032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/3127942540 2024-02-10T09:21:33.791 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:33.790+0000 7ffa30faf700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-10T09:21:34.591 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:34.591+0000 7ffa30faf700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 msgr2=0x7ffa14032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/3127942540 2024-02-10T09:21:34.592 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:34.591+0000 7ffa30faf700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-10T09:21:35.652 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:35.651+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7ffa280257e0 con 0x7ffa2c070430 2024-02-10T09:21:35.652 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:35.651+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 msgr2=0x7ffa14032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T09:21:35.653 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:35.651+0000 7ffa19ffb700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:21:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.654+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ffa28024ca0 con 0x7ffa2c070430 2024-02-10T09:21:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.654+0000 7ffa19ffb700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ffa14033790 0x7ffa14035b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.654+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa2c059100 con 0x7ffa14033790 2024-02-10T09:21:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.655+0000 7ffa30faf700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ffa14033790 0x7ffa14035b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:36.657 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.656+0000 7ffa30faf700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ffa14033790 0x7ffa14035b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ffa1c0036d0 tx=0x7ffa1c006d20).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:36.658 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.656+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ffa2c059100 con 0x7ffa14033790 2024-02-10T09:21:36.668 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.662+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ffa2c045ad0 con 0x7ffa14033790 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa19ffb700 1 -- 172.21.15.83:0/679279261 <== mgr.14114 v2:172.21.15.83:6800/89965517 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ffa2c045ad0 con 0x7ffa14033790 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ffa14033790 msgr2=0x7ffa14035b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ffa14033790 0x7ffa14035b80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ffa1c0036d0 tx=0x7ffa1c006d20).stop 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 msgr2=0x7ffa2c10ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c10ab40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ffa28006800 tx=0x7ffa28006ee0).stop 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 shutdown_connections 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ffa14033790 0x7ffa14035b80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa2c070430 0x7ffa2c10ab40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 --2- 172.21.15.83:0/679279261 >> [v2:172.21.15.83:6800/3127942540,v1:172.21.15.83:6801/3127942540] conn(0x7ffa1402fe70 0x7ffa14032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 >> 172.21.15.83:0/679279261 conn(0x7ffa2c0f6530 msgr2=0x7ffa2c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:36.669 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 shutdown_connections 2024-02-10T09:21:36.670 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:36.663+0000 7ffa33a14700 1 -- 172.21.15.83:0/679279261 wait complete. 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:35.613338+0000 mon.smithi083 (mon.0) 29 : cluster [INF] Active manager daemon smithi083.dowdym restarted 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:35.613714+0000 mon.smithi083 (mon.0) 30 : cluster [INF] Activating manager daemon smithi083.dowdym 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:35.615477+0000 mon.smithi083 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:35.653137+0000 mon.smithi083 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi083.dowdym(active, starting, since 0.0395133s) 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.654059+0000 mon.smithi083 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.654635+0000 mon.smithi083 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mgr metadata", "who": "smithi083.dowdym", "id": "smithi083.dowdym"}]: dispatch 2024-02-10T09:21:37.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.655417+0000 mon.smithi083 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.655591+0000 mon.smithi083 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.655726+0000 mon.smithi083 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:35.669051+0000 mon.smithi083 (mon.0) 38 : cluster [INF] Manager daemon smithi083.dowdym is now available 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.682169+0000 mon.smithi083 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.684728+0000 mon.smithi083 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.687111+0000 mon.smithi083 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.774333+0000 mon.smithi083 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.775058+0000 mon.smithi083 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.776834+0000 mon.smithi083 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.781583+0000 mon.smithi083 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.787869+0000 mon.smithi083 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:21:37.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:35.788988+0000 mon.smithi083 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:21:38.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:36.657895+0000 mgr.smithi083.dowdym (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T09:21:38.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:36.658950+0000 mon.smithi083 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi083.dowdym(active, since 1.04532s) 2024-02-10T09:21:38.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:36.664653+0000 mgr.smithi083.dowdym (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T09:21:39.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:38 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:37.696767+0000 mon.smithi083 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi083.dowdym(active, since 2s) 2024-02-10T09:21:39.781 INFO:teuthology.orchestra.run.smithi083.stderr:mgr epoch 5 is available 2024-02-10T09:21:39.782 INFO:teuthology.orchestra.run.smithi083.stderr:Setting orchestrator backend to cephadm... 2024-02-10T09:21:39.782 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-10T09:21:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.308+0000 7f2324d33700 1 Processor -- start 2024-02-10T09:21:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.308+0000 7f2324d33700 1 -- start start 2024-02-10T09:21:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.308+0000 7f2324d33700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f23200fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.308+0000 7f2324d33700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23200fb6a0 con 0x7f23200fad90 2024-02-10T09:21:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f23200fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:40.314 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f23200fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52690/0 (socket says 172.21.15.83:52690) 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231e59c700 1 -- 172.21.15.83:0/2898092724 learned_addr learned my addr 172.21.15.83:0/2898092724 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231e59c700 1 -- 172.21.15.83:0/2898092724 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200fb7e0 con 0x7f23200fad90 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231e59c700 1 --2- 172.21.15.83:0/2898092724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f23200fb1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f231400b5f0 tx=0x7f2314006ee0).ready entity=mon.0 client_cookie=6a8dbed8d5a3a99f server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231d59a700 1 -- 172.21.15.83:0/2898092724 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231400d600 con 0x7f23200fad90 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231d59a700 1 -- 172.21.15.83:0/2898092724 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f231400dbf0 con 0x7f23200fad90 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.309+0000 7f231d59a700 1 -- 172.21.15.83:0/2898092724 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2314011cb0 con 0x7f23200fad90 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 -- 172.21.15.83:0/2898092724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 msgr2=0x7f23200fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 --2- 172.21.15.83:0/2898092724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f23200fb1b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f231400b5f0 tx=0x7f2314006ee0).stop 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 -- 172.21.15.83:0/2898092724 shutdown_connections 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 --2- 172.21.15.83:0/2898092724 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f23200fb1b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 -- 172.21.15.83:0/2898092724 >> 172.21.15.83:0/2898092724 conn(0x7f23200f6530 msgr2=0x7f23200f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 -- 172.21.15.83:0/2898092724 shutdown_connections 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 -- 172.21.15.83:0/2898092724 wait complete. 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 Processor -- start 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.310+0000 7f2324d33700 1 -- start start 2024-02-10T09:21:40.315 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f2324d33700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f232010ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f2324d33700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232010b070 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f231e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f232010ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f231e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f232010ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52696/0 (socket says 172.21.15.83:52696) 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f231e59c700 1 -- 172.21.15.83:0/434404886 learned_addr learned my addr 172.21.15.83:0/434404886 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f231e59c700 1 -- 172.21.15.83:0/434404886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231400b200 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f231e59c700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f232010ab80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2314002f40 tx=0x7f2314006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231401b410 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f231401ba00 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232010b270 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231401a900 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.311+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232010b700 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.312+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f231400f070 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.312+0000 7f230effd700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f230402fe10 0x7f23040322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.312+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2314041320 con 0x7f23200fad90 2024-02-10T09:21:40.316 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.312+0000 7f231dd9b700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f230402fe10 0x7f23040322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.313+0000 7f231dd9b700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f230402fe10 0x7f23040322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f23100096a0 tx=0x7f2310006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:40.317 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.313+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2320045ad0 con 0x7f23200fad90 2024-02-10T09:21:40.318 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.317+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2314016070 con 0x7f23200fad90 2024-02-10T09:21:40.496 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.495+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2320104d00 con 0x7f230402fe10 2024-02-10T09:21:40.501 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.500+0000 7f230effd700 1 -- 172.21.15.83:0/434404886 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2320104d00 con 0x7f230402fe10 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f230402fe10 msgr2=0x7f23040322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f230402fe10 0x7f23040322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f23100096a0 tx=0x7f2310006b40).stop 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 msgr2=0x7f232010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f232010ab80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f2314002f40 tx=0x7f2314006ee0).stop 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 shutdown_connections 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f230402fe10 0x7f23040322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 --2- 172.21.15.83:0/434404886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23200fad90 0x7f232010ab80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.501+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 >> 172.21.15.83:0/434404886 conn(0x7f23200f6530 msgr2=0x7f23200700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.502+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 shutdown_connections 2024-02-10T09:21:40.504 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:40.502+0000 7f2324d33700 1 -- 172.21.15.83:0/434404886 wait complete. 2024-02-10T09:21:41.062 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f458ba98700 1 Processor -- start 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f458ba98700 1 -- start start 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f458ba98700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f45840fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f458ba98700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840fef40 con 0x7f4584070430 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f4589834700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f45840fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f4589834700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f45840fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52704/0 (socket says 172.21.15.83:52704) 2024-02-10T09:21:41.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.598+0000 7f4589834700 1 -- 172.21.15.83:0/383409464 learned_addr learned my addr 172.21.15.83:0/383409464 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.599+0000 7f4589834700 1 -- 172.21.15.83:0/383409464 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45840ff080 con 0x7f4584070430 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.599+0000 7f4589834700 1 --2- 172.21.15.83:0/383409464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f45840fea50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f457400b5f0 tx=0x7f4574006ee0).ready entity=mon.0 client_cookie=4a584af2abe11474 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.599+0000 7f4588832700 1 -- 172.21.15.83:0/383409464 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457400d600 con 0x7f4584070430 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.599+0000 7f4588832700 1 -- 172.21.15.83:0/383409464 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f457400dbf0 con 0x7f4584070430 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.599+0000 7f4588832700 1 -- 172.21.15.83:0/383409464 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4574011cb0 con 0x7f4584070430 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 -- 172.21.15.83:0/383409464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 msgr2=0x7f45840fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 --2- 172.21.15.83:0/383409464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f45840fea50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f457400b5f0 tx=0x7f4574006ee0).stop 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 -- 172.21.15.83:0/383409464 shutdown_connections 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 --2- 172.21.15.83:0/383409464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f45840fea50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 -- 172.21.15.83:0/383409464 >> 172.21.15.83:0/383409464 conn(0x7f45840f6530 msgr2=0x7f45840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 -- 172.21.15.83:0/383409464 shutdown_connections 2024-02-10T09:21:41.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.600+0000 7f458ba98700 1 -- 172.21.15.83:0/383409464 wait complete. 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f458ba98700 1 Processor -- start 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f458ba98700 1 -- start start 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f458ba98700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f458410ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f458ba98700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458410b030 con 0x7f4584070430 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f4589834700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f458410ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f4589834700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f458410ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52712/0 (socket says 172.21.15.83:52712) 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f4589834700 1 -- 172.21.15.83:0/2243504214 learned_addr learned my addr 172.21.15.83:0/2243504214 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f4589834700 1 -- 172.21.15.83:0/2243504214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457400b200 con 0x7f4584070430 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f4589834700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f458410ab40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4574006800 tx=0x7f4574006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457401b570 con 0x7f4584070430 2024-02-10T09:21:41.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458410b230 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f457401bb60 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.601+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f457401a9f0 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.602+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458410b6c0 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.602+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f457400f070 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.602+0000 7f45727fc700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f456c02fe10 0x7f456c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.602+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4574041450 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.602+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4584045ad0 con 0x7f4584070430 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.603+0000 7f4589033700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f456c02fe10 0x7f456c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:41.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.603+0000 7f4589033700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f456c02fe10 0x7f456c0322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f45800096a0 tx=0x7f4580006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:41.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.607+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4574016070 con 0x7f4584070430 2024-02-10T09:21:41.760 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.758+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f458410bf20 con 0x7f456c02fe10 2024-02-10T09:21:41.760 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.759+0000 7f45727fc700 1 -- 172.21.15.83:0/2243504214 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f458410bf20 con 0x7f456c02fe10 2024-02-10T09:21:41.760 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: value unchanged 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f456c02fe10 msgr2=0x7f456c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f456c02fe10 0x7f456c0322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f45800096a0 tx=0x7f4580006b40).stop 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 msgr2=0x7f458410ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f458410ab40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4574006800 tx=0x7f4574006ee0).stop 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 shutdown_connections 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f456c02fe10 0x7f456c0322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 --2- 172.21.15.83:0/2243504214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4584070430 0x7f458410ab40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 >> 172.21.15.83:0/2243504214 conn(0x7f45840f6530 msgr2=0x7f45840f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 shutdown_connections 2024-02-10T09:21:41.763 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:41.760+0000 7f458ba98700 1 -- 172.21.15.83:0/2243504214 wait complete. 2024-02-10T09:21:41.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:40.497469+0000 mgr.smithi083.dowdym (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:41.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:40.501237+0000 mon.smithi083 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:41.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:40.504443+0000 mon.smithi083 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:42.336 INFO:teuthology.orchestra.run.smithi083.stderr:Generating ssh key... 2024-02-10T09:21:42.336 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-10T09:21:42.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:42 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:41.760508+0000 mgr.smithi083.dowdym (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.793+0000 7ff475bc8700 1 Processor -- start 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.793+0000 7ff475bc8700 1 -- start start 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.793+0000 7ff475bc8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff470100f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.793+0000 7ff475bc8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4700fc970 con 0x7ff470070420 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff470100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff470100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52718/0 (socket says 172.21.15.83:52718) 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46f7fe700 1 -- 172.21.15.83:0/1102172234 learned_addr learned my addr 172.21.15.83:0/1102172234 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46f7fe700 1 -- 172.21.15.83:0/1102172234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470101470 con 0x7ff470070420 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46f7fe700 1 --2- 172.21.15.83:0/1102172234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff470100f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff46000b5f0 tx=0x7ff460006ee0).ready entity=mon.0 client_cookie=6ff29dac82d29470 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46e7fc700 1 -- 172.21.15.83:0/1102172234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46000d600 con 0x7ff470070420 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46e7fc700 1 -- 172.21.15.83:0/1102172234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff46000dbf0 con 0x7ff470070420 2024-02-10T09:21:42.809 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.794+0000 7ff46e7fc700 1 -- 172.21.15.83:0/1102172234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff460011cb0 con 0x7ff470070420 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- 172.21.15.83:0/1102172234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 msgr2=0x7ff470100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 --2- 172.21.15.83:0/1102172234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff470100f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff46000b5f0 tx=0x7ff460006ee0).stop 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- 172.21.15.83:0/1102172234 shutdown_connections 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 --2- 172.21.15.83:0/1102172234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff470100f80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- 172.21.15.83:0/1102172234 >> 172.21.15.83:0/1102172234 conn(0x7ff4700f65c0 msgr2=0x7ff4700f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- 172.21.15.83:0/1102172234 shutdown_connections 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- 172.21.15.83:0/1102172234 wait complete. 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 Processor -- start 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- start start 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff47010cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.795+0000 7ff475bc8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47010d2b0 con 0x7ff470070420 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff46f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff47010cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff46f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff47010cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:52720/0 (socket says 172.21.15.83:52720) 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff46f7fe700 1 -- 172.21.15.83:0/3118914258 learned_addr learned my addr 172.21.15.83:0/3118914258 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:42.810 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff46f7fe700 1 -- 172.21.15.83:0/3118914258 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46000b200 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff46f7fe700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff47010cdc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff460006800 tx=0x7ff460006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46001b570 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff46001bb60 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47010d4b0 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff46001a9f0 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.796+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47010e0f0 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.797+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff46000f070 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.797+0000 7ff457fff700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff45c02fe10 0x7ff45c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.797+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff4600414a0 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.797+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47010d940 con 0x7ff470070420 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.797+0000 7ff46effd700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff45c02fe10 0x7ff45c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.798+0000 7ff46effd700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff45c02fe10 0x7ff45c0322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff4640096a0 tx=0x7ff464006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:42.811 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.801+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff460016070 con 0x7ff470070420 2024-02-10T09:21:42.952 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:42.950+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4700008d0 con 0x7ff45c02fe10 2024-02-10T09:21:43.096 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.094+0000 7ff457fff700 1 -- 172.21.15.83:0/3118914258 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff4700008d0 con 0x7ff45c02fe10 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff45c02fe10 msgr2=0x7ff45c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff45c02fe10 0x7ff45c0322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff4640096a0 tx=0x7ff464006b40).stop 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 msgr2=0x7ff47010cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff47010cdc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff460006800 tx=0x7ff460006ee0).stop 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 shutdown_connections 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff45c02fe10 0x7ff45c0322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 --2- 172.21.15.83:0/3118914258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff470070420 0x7ff47010cdc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 >> 172.21.15.83:0/3118914258 conn(0x7ff4700f65c0 msgr2=0x7ff4700f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 shutdown_connections 2024-02-10T09:21:43.099 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:43.096+0000 7ff475bc8700 1 -- 172.21.15.83:0/3118914258 wait complete. 2024-02-10T09:21:43.776 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-10T09:21:44.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.308+0000 7f6ef89b2700 1 Processor -- start 2024-02-10T09:21:44.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.308+0000 7f6ef89b2700 1 -- start start 2024-02-10T09:21:44.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef89b2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:44.355 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef89b2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef40fc8a0 con 0x7f6ef40703d0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43752/0 (socket says 172.21.15.83:43752) 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef259c700 1 -- 172.21.15.83:0/2096207022 learned_addr learned my addr 172.21.15.83:0/2096207022 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef259c700 1 -- 172.21.15.83:0/2096207022 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef41013a0 con 0x7f6ef40703d0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef259c700 1 --2- 172.21.15.83:0/2096207022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef4100eb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6edc00b5f0 tx=0x7f6edc006ee0).ready entity=mon.0 client_cookie=764a6b5e46762666 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef159a700 1 -- 172.21.15.83:0/2096207022 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc00d600 con 0x7f6ef40703d0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef159a700 1 -- 172.21.15.83:0/2096207022 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6edc00dbf0 con 0x7f6ef40703d0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.309+0000 7f6ef159a700 1 -- 172.21.15.83:0/2096207022 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc011cb0 con 0x7f6ef40703d0 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 -- 172.21.15.83:0/2096207022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 msgr2=0x7f6ef4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 --2- 172.21.15.83:0/2096207022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef4100eb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6edc00b5f0 tx=0x7f6edc006ee0).stop 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 -- 172.21.15.83:0/2096207022 shutdown_connections 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 --2- 172.21.15.83:0/2096207022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef4100eb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 -- 172.21.15.83:0/2096207022 >> 172.21.15.83:0/2096207022 conn(0x7f6ef40f6530 msgr2=0x7f6ef40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 -- 172.21.15.83:0/2096207022 shutdown_connections 2024-02-10T09:21:44.356 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.310+0000 7f6ef89b2700 1 -- 172.21.15.83:0/2096207022 wait complete. 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef89b2700 1 Processor -- start 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef89b2700 1 -- start start 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef89b2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef410ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef89b2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef410d310 con 0x7f6ef40703d0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef410ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef410ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43758/0 (socket says 172.21.15.83:43758) 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef259c700 1 -- 172.21.15.83:0/1577521014 learned_addr learned my addr 172.21.15.83:0/1577521014 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef259c700 1 -- 172.21.15.83:0/1577521014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc00b200 con 0x7f6ef40703d0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef259c700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef410ce20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f6edc006c70 tx=0x7f6edc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc011400 con 0x7f6ef40703d0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6edc01b8a0 con 0x7f6ef40703d0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6edc01a7c0 con 0x7f6ef40703d0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef410d510 con 0x7f6ef40703d0 2024-02-10T09:21:44.357 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.311+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef410e120 con 0x7f6ef40703d0 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.312+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6edc00f070 con 0x7f6ef40703d0 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.312+0000 7f6eeaffd700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f6ed402fe70 0x7f6ed4032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.312+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef4059100 con 0x7f6ef40703d0 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.312+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6edc041310 con 0x7f6ef40703d0 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.312+0000 7f6ef1d9b700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f6ed402fe70 0x7f6ed4032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.313+0000 7f6ef1d9b700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f6ed402fe70 0x7f6ed4032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f6ee40096a0 tx=0x7f6ee4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:44.358 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.316+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6edc016080 con 0x7f6ef40703d0 2024-02-10T09:21:44.467 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.466+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ef410e360 con 0x7f6ed402fe70 2024-02-10T09:21:44.468 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.467+0000 7f6eeaffd700 1 -- 172.21.15.83:0/1577521014 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f6ef410e360 con 0x7f6ed402fe70 2024-02-10T09:21:44.468 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV2fnWQMWRK1PgFJGstXW857w0XOMxdml4eb4+5LDxzWZL8ID5Fd4Av/O6J8X1goewjCM0qJS9VQIb94aIlSlmjyqagMsIeLHxPrq8lCmAUwNFW6Px28uMVx3a5S8+PLDigtyK5yjhC8PNlyWdJ7GyZ94JGm3JC1S3rCobvsrZ1raKGnDt69cW7UGiciWHfDoOEyUoWiP+HfyCLeWQh8HHAh76UdjrN0QS8PHbzfryA0igmdJZdrT4KdLp6PX40JtFefABlcPGZPPQZNV96gODtbk/Dq6hEkITgh+K9jDj3eNtpAkth4ufO9Ru9L/PdkQFnrRDKFzjFXjmPnxVeVu0UFm8x6JV69K262ltlPi8z1k8a68Ab+GCSyxlEHpIqoKNJ80o+xMhn4eJD/PjdMiXGh6WTF4fV2BfOuHaW1Jit8hu7gczCFOt9lP0hoBtTGZpA73qGz4cmHUwqaGgdLUBo+pSoMlpSKWP6IoxfRL6n7z5eI8rfRv9PMZ2bBAy8R0= ceph-a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:21:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f6ed402fe70 msgr2=0x7f6ed4032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:44.470 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f6ed402fe70 0x7f6ed4032330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f6ee40096a0 tx=0x7f6ee4006b40).stop 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 msgr2=0x7f6ef410ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef410ce20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f6edc006c70 tx=0x7f6edc006ee0).stop 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 shutdown_connections 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f6ed402fe70 0x7f6ed4032330 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 --2- 172.21.15.83:0/1577521014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ef40703d0 0x7f6ef410ce20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 >> 172.21.15.83:0/1577521014 conn(0x7f6ef40f6530 msgr2=0x7f6ef40f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 shutdown_connections 2024-02-10T09:21:44.471 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:44.468+0000 7f6ef89b2700 1 -- 172.21.15.83:0/1577521014 wait complete. 2024-02-10T09:21:44.493 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:42.952818+0000 mgr.smithi083.dowdym (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-10T09:21:44.493 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:43.093724+0000 mon.smithi083 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:44.493 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:43.095472+0000 mon.smithi083 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:45.299 INFO:teuthology.orchestra.run.smithi083.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:21:45.300 INFO:teuthology.orchestra.run.smithi083.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T09:21:45.300 INFO:teuthology.orchestra.run.smithi083.stderr:Adding host smithi083... 2024-02-10T09:21:45.300 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi083 172.21.15.83 2024-02-10T09:21:45.896 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98e59b3700 1 Processor -- start 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98e59b3700 1 -- start start 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98e59b3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e00fa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98e59b3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e00fae40 con 0x7f98e00fa530 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98deffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e00fa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98deffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e00fa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43770/0 (socket says 172.21.15.83:43770) 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.851+0000 7f98deffd700 1 -- 172.21.15.83:0/4078292932 learned_addr learned my addr 172.21.15.83:0/4078292932 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98deffd700 1 -- 172.21.15.83:0/4078292932 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e00faf80 con 0x7f98e00fa530 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98deffd700 1 --2- 172.21.15.83:0/4078292932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e00fa950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f98d000b5f0 tx=0x7f98d0006ee0).ready entity=mon.0 client_cookie=ab5e9f7df4fa5165 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98ddffb700 1 -- 172.21.15.83:0/4078292932 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98d000d600 con 0x7f98e00fa530 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98ddffb700 1 -- 172.21.15.83:0/4078292932 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f98d000dbf0 con 0x7f98e00fa530 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98ddffb700 1 -- 172.21.15.83:0/4078292932 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98d0011cb0 con 0x7f98e00fa530 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 -- 172.21.15.83:0/4078292932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 msgr2=0x7f98e00fa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 --2- 172.21.15.83:0/4078292932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e00fa950 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f98d000b5f0 tx=0x7f98d0006ee0).stop 2024-02-10T09:21:45.897 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 -- 172.21.15.83:0/4078292932 shutdown_connections 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 --2- 172.21.15.83:0/4078292932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e00fa950 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 -- 172.21.15.83:0/4078292932 >> 172.21.15.83:0/4078292932 conn(0x7f98e00f5d30 msgr2=0x7f98e00f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 -- 172.21.15.83:0/4078292932 shutdown_connections 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.852+0000 7f98e59b3700 1 -- 172.21.15.83:0/4078292932 wait complete. 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98e59b3700 1 Processor -- start 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98e59b3700 1 -- start start 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98e59b3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e0109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98e59b3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e0109f90 con 0x7f98e00fa530 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98deffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e0109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98deffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e0109aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43772/0 (socket says 172.21.15.83:43772) 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.853+0000 7f98deffd700 1 -- 172.21.15.83:0/743048369 learned_addr learned my addr 172.21.15.83:0/743048369 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98deffd700 1 -- 172.21.15.83:0/743048369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d000b200 con 0x7f98e00fa530 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98deffd700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e0109aa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f98d000bf40 tx=0x7f98d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:45.898 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98d001b570 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f98d001bb60 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e010a190 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98d001aa20 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e010a620 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.854+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f98d000f070 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.855+0000 7f98c77fe700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f98cc02fe10 0x7f98cc0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.855+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e0045ad0 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.855+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f98d00414d0 con 0x7f98e00fa530 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.855+0000 7f98de7fc700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f98cc02fe10 0x7f98cc0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.855+0000 7f98de7fc700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f98cc02fe10 0x7f98cc0322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f98d40096a0 tx=0x7f98d4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:45.899 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:45.859+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98d0016080 con 0x7f98e00fa530 2024-02-10T09:21:46.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:44.468407+0000 mgr.smithi083.dowdym (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:46.031 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.030+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}) v1 -- 0x7f98e010ae80 con 0x7f98cc02fe10 2024-02-10T09:21:46.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Added host 'smithi083' 2024-02-10T09:21:46.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.626+0000 7f98c77fe700 1 -- 172.21.15.83:0/743048369 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f98e010ae80 con 0x7f98cc02fe10 2024-02-10T09:21:46.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.628+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f98cc02fe10 msgr2=0x7f98cc0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:46.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.628+0000 7f98e59b3700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f98cc02fe10 0x7f98cc0322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f98d40096a0 tx=0x7f98d4006b40).stop 2024-02-10T09:21:46.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 msgr2=0x7f98e0109aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:46.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e0109aa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f98d000bf40 tx=0x7f98d0006ee0).stop 2024-02-10T09:21:46.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 shutdown_connections 2024-02-10T09:21:46.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f98cc02fe10 0x7f98cc0322d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:46.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 --2- 172.21.15.83:0/743048369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f98e00fa530 0x7f98e0109aa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:46.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 >> 172.21.15.83:0/743048369 conn(0x7f98e00f5d30 msgr2=0x7f98e0100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:46.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 shutdown_connections 2024-02-10T09:21:46.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:46.629+0000 7f98e59b3700 1 -- 172.21.15.83:0/743048369 wait complete. 2024-02-10T09:21:47.123 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying mon service with default placement... 2024-02-10T09:21:47.123 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-10T09:21:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95ce8ad700 1 Processor -- start 2024-02-10T09:21:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95ce8ad700 1 -- start start 2024-02-10T09:21:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95ce8ad700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95ce8ad700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c80fef40 con 0x7f95c8070430 2024-02-10T09:21:47.637 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95c7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95c7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43782/0 (socket says 172.21.15.83:43782) 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95c7fff700 1 -- 172.21.15.83:0/4056350169 learned_addr learned my addr 172.21.15.83:0/4056350169 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95c7fff700 1 -- 172.21.15.83:0/4056350169 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c80ff080 con 0x7f95c8070430 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.631+0000 7f95c7fff700 1 --2- 172.21.15.83:0/4056350169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f95b800b5f0 tx=0x7f95b8006ee0).ready entity=mon.0 client_cookie=eed4d4faf285c8ac server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95c6ffd700 1 -- 172.21.15.83:0/4056350169 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b800d600 con 0x7f95c8070430 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95c6ffd700 1 -- 172.21.15.83:0/4056350169 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f95b800dbf0 con 0x7f95c8070430 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95c6ffd700 1 -- 172.21.15.83:0/4056350169 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b8011cb0 con 0x7f95c8070430 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95ce8ad700 1 -- 172.21.15.83:0/4056350169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 msgr2=0x7f95c80fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95ce8ad700 1 --2- 172.21.15.83:0/4056350169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f95b800b5f0 tx=0x7f95b8006ee0).stop 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95ce8ad700 1 -- 172.21.15.83:0/4056350169 shutdown_connections 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95ce8ad700 1 --2- 172.21.15.83:0/4056350169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fea50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.632+0000 7f95ce8ad700 1 -- 172.21.15.83:0/4056350169 >> 172.21.15.83:0/4056350169 conn(0x7f95c80f6530 msgr2=0x7f95c80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95ce8ad700 1 -- 172.21.15.83:0/4056350169 shutdown_connections 2024-02-10T09:21:47.638 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95ce8ad700 1 -- 172.21.15.83:0/4056350169 wait complete. 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95ce8ad700 1 Processor -- start 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95ce8ad700 1 -- start start 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95ce8ad700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95ce8ad700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c80fa950 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.633+0000 7f95c7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43784/0 (socket says 172.21.15.83:43784) 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c7fff700 1 -- 172.21.15.83:0/2349100638 learned_addr learned my addr 172.21.15.83:0/2349100638 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c7fff700 1 -- 172.21.15.83:0/2349100638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b800b200 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c7fff700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fc4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f95b8006800 tx=0x7f95b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b801b570 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f95b801bb60 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b801a9f0 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c80faaf0 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.634+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c80faf80 con 0x7f95c8070430 2024-02-10T09:21:47.639 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.635+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f95b800f070 con 0x7f95c8070430 2024-02-10T09:21:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.635+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c8045ad0 con 0x7f95c8070430 2024-02-10T09:21:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.635+0000 7f95c4ff9700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f95b402fe70 0x7f95b4032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.635+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f95b8041360 con 0x7f95c8070430 2024-02-10T09:21:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.635+0000 7f95c77fe700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f95b402fe70 0x7f95b4032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.636+0000 7f95c77fe700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f95b402fe70 0x7f95b4032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f95bc0096a0 tx=0x7f95bc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:47.640 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.639+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95b8022d70 con 0x7f95c8070430 2024-02-10T09:21:47.815 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.813+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f95c80fb1c0 con 0x7f95b402fe70 2024-02-10T09:21:47.820 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.818+0000 7f95c4ff9700 1 -- 172.21.15.83:0/2349100638 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f95c80fb1c0 con 0x7f95b402fe70 2024-02-10T09:21:47.820 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T09:21:47.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f95b402fe70 msgr2=0x7f95b4032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:47.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f95b402fe70 0x7f95b4032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f95bc0096a0 tx=0x7f95bc006b40).stop 2024-02-10T09:21:47.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 msgr2=0x7f95c80fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:47.822 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fc4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f95b8006800 tx=0x7f95b8006ee0).stop 2024-02-10T09:21:47.823 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 shutdown_connections 2024-02-10T09:21:47.823 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f95b402fe70 0x7f95b4032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:47.823 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 --2- 172.21.15.83:0/2349100638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f95c8070430 0x7f95c80fc4d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:47.823 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 >> 172.21.15.83:0/2349100638 conn(0x7f95c80f6530 msgr2=0x7f95c80f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:47.823 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 shutdown_connections 2024-02-10T09:21:47.823 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:47.820+0000 7f95ce8ad700 1 -- 172.21.15.83:0/2349100638 wait complete. 2024-02-10T09:21:48.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:46.032415+0000 mgr.smithi083.dowdym (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "addr": "172.21.15.83", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:48.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:46.627271+0000 mon.smithi083 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:48.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:46.627588+0000 mgr.smithi083.dowdym (mgr.14114) 9 : cephadm [INF] Added host smithi083 2024-02-10T09:21:48.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:46.627808+0000 mon.smithi083 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:48.362 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying mgr service with default placement... 2024-02-10T09:21:48.363 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-10T09:21:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.843+0000 7f7704e69700 1 Processor -- start 2024-02-10T09:21:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.843+0000 7f7704e69700 1 -- start start 2024-02-10T09:21:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.843+0000 7f7704e69700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f77000feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.843+0000 7f7704e69700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000fefa0 con 0x7f77000703d0 2024-02-10T09:21:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fe59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f77000feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:48.849 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fe59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f77000feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43800/0 (socket says 172.21.15.83:43800) 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fe59c700 1 -- 172.21.15.83:0/2250509937 learned_addr learned my addr 172.21.15.83:0/2250509937 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fe59c700 1 -- 172.21.15.83:0/2250509937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77000ff0e0 con 0x7f77000703d0 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fe59c700 1 --2- 172.21.15.83:0/2250509937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f77000feab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f76f400b5f0 tx=0x7f76f4006ee0).ready entity=mon.0 client_cookie=eef06d321963edae server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fd59a700 1 -- 172.21.15.83:0/2250509937 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f400d600 con 0x7f77000703d0 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fd59a700 1 -- 172.21.15.83:0/2250509937 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f76f400dbf0 con 0x7f77000703d0 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.844+0000 7f76fd59a700 1 -- 172.21.15.83:0/2250509937 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f4011cb0 con 0x7f77000703d0 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- 172.21.15.83:0/2250509937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 msgr2=0x7f77000feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 --2- 172.21.15.83:0/2250509937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f77000feab0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f76f400b5f0 tx=0x7f76f4006ee0).stop 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- 172.21.15.83:0/2250509937 shutdown_connections 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 --2- 172.21.15.83:0/2250509937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f77000feab0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- 172.21.15.83:0/2250509937 >> 172.21.15.83:0/2250509937 conn(0x7f77000f6530 msgr2=0x7f77000f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- 172.21.15.83:0/2250509937 shutdown_connections 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- 172.21.15.83:0/2250509937 wait complete. 2024-02-10T09:21:48.850 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 Processor -- start 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- start start 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f770010ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.845+0000 7f7704e69700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770010b060 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76fe59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f770010ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76fe59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f770010ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43804/0 (socket says 172.21.15.83:43804) 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76fe59c700 1 -- 172.21.15.83:0/1060761816 learned_addr learned my addr 172.21.15.83:0/1060761816 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76fe59c700 1 -- 172.21.15.83:0/1060761816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f400b200 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76fe59c700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f770010ab70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f76f4006a20 tx=0x7f76f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f401b570 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f76f401bb60 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770010b260 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f401aa20 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.846+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770010b6f0 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.847+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f76f400f070 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.847+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7700045ad0 con 0x7f77000703d0 2024-02-10T09:21:48.851 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.847+0000 7f76eeffd700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f76e402fe10 0x7f76e40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.847+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f76f4041360 con 0x7f77000703d0 2024-02-10T09:21:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.847+0000 7f76fdd9b700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f76e402fe10 0x7f76e40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:48.852 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.848+0000 7f76fdd9b700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f76e402fe10 0x7f76e40322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f76f00096a0 tx=0x7f76f0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:48.853 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:48.851+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76f401a370 con 0x7f77000703d0 2024-02-10T09:21:49.028 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.026+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f770010bf50 con 0x7f76e402fe10 2024-02-10T09:21:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.031+0000 7f76eeffd700 1 -- 172.21.15.83:0/1060761816 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f770010bf50 con 0x7f76e402fe10 2024-02-10T09:21:49.032 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T09:21:49.034 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f76e402fe10 msgr2=0x7f76e40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f76e402fe10 0x7f76e40322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f76f00096a0 tx=0x7f76f0006b40).stop 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 msgr2=0x7f770010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f770010ab70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f76f4006a20 tx=0x7f76f4006ee0).stop 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 shutdown_connections 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f76e402fe10 0x7f76e40322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 --2- 172.21.15.83:0/1060761816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77000703d0 0x7f770010ab70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 >> 172.21.15.83:0/1060761816 conn(0x7f77000f6530 msgr2=0x7f77000f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 shutdown_connections 2024-02-10T09:21:49.035 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:49.032+0000 7f7704e69700 1 -- 172.21.15.83:0/1060761816 wait complete. 2024-02-10T09:21:49.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:47.816159+0000 mgr.smithi083.dowdym (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:49.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:47.816849+0000 mgr.smithi083.dowdym (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-10T09:21:49.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:47.819783+0000 mon.smithi083 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:49.728 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying crash service with default placement... 2024-02-10T09:21:49.728 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-10T09:21:50.237 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.234+0000 7ff31950a700 1 Processor -- start 2024-02-10T09:21:50.237 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.234+0000 7ff31950a700 1 -- start start 2024-02-10T09:21:50.237 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.234+0000 7ff31950a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:50.237 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.234+0000 7ff31950a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140fef40 con 0x7ff314070430 2024-02-10T09:21:50.237 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff312ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:50.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff312ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43810/0 (socket says 172.21.15.83:43810) 2024-02-10T09:21:50.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff312ffd700 1 -- 172.21.15.83:0/3316796318 learned_addr learned my addr 172.21.15.83:0/3316796318 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:50.238 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff312ffd700 1 -- 172.21.15.83:0/3316796318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3140ff080 con 0x7ff314070430 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff312ffd700 1 --2- 172.21.15.83:0/3316796318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fea50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff30800b5f0 tx=0x7ff308006ee0).ready entity=mon.0 client_cookie=22f45bbf2595e95d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff311ffb700 1 -- 172.21.15.83:0/3316796318 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30800d600 con 0x7ff314070430 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff311ffb700 1 -- 172.21.15.83:0/3316796318 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff30800dbf0 con 0x7ff314070430 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.235+0000 7ff311ffb700 1 -- 172.21.15.83:0/3316796318 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff308011cb0 con 0x7ff314070430 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 -- 172.21.15.83:0/3316796318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 msgr2=0x7ff3140fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 --2- 172.21.15.83:0/3316796318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fea50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff30800b5f0 tx=0x7ff308006ee0).stop 2024-02-10T09:21:50.240 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 -- 172.21.15.83:0/3316796318 shutdown_connections 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 --2- 172.21.15.83:0/3316796318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fea50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 -- 172.21.15.83:0/3316796318 >> 172.21.15.83:0/3316796318 conn(0x7ff3140f6530 msgr2=0x7ff3140f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 -- 172.21.15.83:0/3316796318 shutdown_connections 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 -- 172.21.15.83:0/3316796318 wait complete. 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 Processor -- start 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.236+0000 7ff31950a700 1 -- start start 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff31950a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff31950a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3140fa950 con 0x7ff314070430 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff312ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff312ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43812/0 (socket says 172.21.15.83:43812) 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff312ffd700 1 -- 172.21.15.83:0/3066616823 learned_addr learned my addr 172.21.15.83:0/3066616823 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff312ffd700 1 -- 172.21.15.83:0/3066616823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30800b200 con 0x7ff314070430 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff312ffd700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fc4d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff308006800 tx=0x7ff308006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:50.241 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30801b570 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff30801bb60 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3140faaf0 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff30801a9f0 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.237+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3140faf80 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.238+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff30800f070 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.238+0000 7ff2fb7fe700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff30002fe10 0x7ff3000322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.238+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff308042370 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.238+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff314045ad0 con 0x7ff314070430 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.238+0000 7ff3127fc700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff30002fe10 0x7ff3000322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:50.242 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.239+0000 7ff3127fc700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff30002fe10 0x7ff3000322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff3040096a0 tx=0x7ff304006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:50.244 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.242+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff308016070 con 0x7ff314070430 2024-02-10T09:21:50.417 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.415+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3140ffe30 con 0x7ff30002fe10 2024-02-10T09:21:50.420 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.419+0000 7ff2fb7fe700 1 -- 172.21.15.83:0/3066616823 <== mgr.14114 v2:172.21.15.83:6800/89965517 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff3140ffe30 con 0x7ff30002fe10 2024-02-10T09:21:50.420 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.420+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff30002fe10 msgr2=0x7ff3000322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.420+0000 7ff31950a700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff30002fe10 0x7ff3000322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff3040096a0 tx=0x7ff304006b40).stop 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.420+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 msgr2=0x7ff3140fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.420+0000 7ff31950a700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fc4d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7ff308006800 tx=0x7ff308006ee0).stop 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.420+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 shutdown_connections 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.421+0000 7ff31950a700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7ff30002fe10 0x7ff3000322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.421+0000 7ff31950a700 1 --2- 172.21.15.83:0/3066616823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff314070430 0x7ff3140fc4d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.421+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 >> 172.21.15.83:0/3066616823 conn(0x7ff3140f6530 msgr2=0x7ff3140f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:50.423 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.421+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 shutdown_connections 2024-02-10T09:21:50.424 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:50.421+0000 7ff31950a700 1 -- 172.21.15.83:0/3066616823 wait complete. 2024-02-10T09:21:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:50 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:49.028926+0000 mgr.smithi083.dowdym (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:50 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:49.029453+0000 mgr.smithi083.dowdym (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-10T09:21:50.445 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:50 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:49.032259+0000 mon.smithi083 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:51.043 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling mgr prometheus module... 2024-02-10T09:21:51.043 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-10T09:21:51.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.601+0000 7f6461f9a700 1 Processor -- start 2024-02-10T09:21:51.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.601+0000 7f6461f9a700 1 -- start start 2024-02-10T09:21:51.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.601+0000 7f6461f9a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.601+0000 7f6461f9a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c0fb6a0 con 0x7f645c0fad90 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43824/0 (socket says 172.21.15.83:43824) 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645b7fe700 1 -- 172.21.15.83:0/3914868947 learned_addr learned my addr 172.21.15.83:0/3914868947 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645b7fe700 1 -- 172.21.15.83:0/3914868947 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c0fb7e0 con 0x7f645c0fad90 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645b7fe700 1 --2- 172.21.15.83:0/3914868947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c0fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f645000b5f0 tx=0x7f6450006ee0).ready entity=mon.0 client_cookie=79ec70adbc03619b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:51.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645a7fc700 1 -- 172.21.15.83:0/3914868947 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f645000d600 con 0x7f645c0fad90 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645a7fc700 1 -- 172.21.15.83:0/3914868947 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f645000dbf0 con 0x7f645c0fad90 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.602+0000 7f645a7fc700 1 -- 172.21.15.83:0/3914868947 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6450011cb0 con 0x7f645c0fad90 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 -- 172.21.15.83:0/3914868947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 msgr2=0x7f645c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 --2- 172.21.15.83:0/3914868947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c0fb1b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f645000b5f0 tx=0x7f6450006ee0).stop 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 -- 172.21.15.83:0/3914868947 shutdown_connections 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 --2- 172.21.15.83:0/3914868947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c0fb1b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 -- 172.21.15.83:0/3914868947 >> 172.21.15.83:0/3914868947 conn(0x7f645c0f6530 msgr2=0x7f645c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 -- 172.21.15.83:0/3914868947 shutdown_connections 2024-02-10T09:21:51.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 -- 172.21.15.83:0/3914868947 wait complete. 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 Processor -- start 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.603+0000 7f6461f9a700 1 -- start start 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6461f9a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6461f9a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c10d2c0 con 0x7f645c0fad90 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f645b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f645b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c10cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:43828/0 (socket says 172.21.15.83:43828) 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f645b7fe700 1 -- 172.21.15.83:0/1632148148 learned_addr learned my addr 172.21.15.83:0/1632148148 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f645b7fe700 1 -- 172.21.15.83:0/1632148148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645000b200 con 0x7f645c0fad90 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f645b7fe700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c10cdd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6450002f40 tx=0x7f6450006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f645001b410 con 0x7f645c0fad90 2024-02-10T09:21:51.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c10d4c0 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f645001ba00 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f645001a900 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.604+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c10e100 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.605+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f645000f070 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.605+0000 7f6443fff700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f644802fe10 0x7f64480322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.605+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f645001bb70 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.605+0000 7f645affd700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f644802fe10 0x7f64480322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.605+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c10d950 con 0x7f645c0fad90 2024-02-10T09:21:51.609 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.606+0000 7f645affd700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f644802fe10 0x7f64480322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f644c0096a0 tx=0x7f644c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:51.611 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.609+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6450022d50 con 0x7f645c0fad90 2024-02-10T09:21:51.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:50.417813+0000 mgr.smithi083.dowdym (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:21:51.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:50.418423+0000 mgr.smithi083.dowdym (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-10T09:21:51.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:50.420559+0000 mon.smithi083 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.83:0/449040367' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:51.798 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:51.797+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f645c0fcb80 con 0x7f645c0fad90 2024-02-10T09:21:52.424 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.422+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f6450025980 con 0x7f645c0fad90 2024-02-10T09:21:52.424 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.423+0000 7f6443fff700 1 -- 172.21.15.83:0/1632148148 <== mon.0 v2:172.21.15.83:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f645002e3e0 con 0x7f645c0fad90 2024-02-10T09:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f644802fe10 msgr2=0x7f64480322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f644802fe10 0x7f64480322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f644c0096a0 tx=0x7f644c006b40).stop 2024-02-10T09:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 msgr2=0x7f645c10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c10cdd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6450002f40 tx=0x7f6450006ee0).stop 2024-02-10T09:21:52.427 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 shutdown_connections 2024-02-10T09:21:52.428 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7f644802fe10 0x7f64480322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:52.428 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 --2- 172.21.15.83:0/1632148148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f645c0fad90 0x7f645c10cdd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:52.428 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 >> 172.21.15.83:0/1632148148 conn(0x7f645c0f6530 msgr2=0x7f645c0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:52.428 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 shutdown_connections 2024-02-10T09:21:52.429 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:52.426+0000 7f6461f9a700 1 -- 172.21.15.83:0/1632148148 wait complete. 2024-02-10T09:21:52.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:52 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:51.798943+0000 mon.smithi083 (mon.0) 59 : audit [INF] from='client.? 172.21.15.83:0/1632148148' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T09:21:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying prometheus service with default placement... 2024-02-10T09:21:53.099 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-10T09:21:53.764 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.724+0000 7fd0d359e700 1 Processor -- start 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.724+0000 7fd0d359e700 1 -- start start 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.724+0000 7fd0d359e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d40684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.724+0000 7fd0d359e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4069770 con 0x7fd0d40680c0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.724+0000 7fd0d259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d40684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d40684e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53318/0 (socket says 172.21.15.83:53318) 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d259c700 1 -- 172.21.15.83:0/2657041263 learned_addr learned my addr 172.21.15.83:0/2657041263 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d259c700 1 -- 172.21.15.83:0/2657041263 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40698b0 con 0x7fd0d40680c0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d259c700 1 --2- 172.21.15.83:0/2657041263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d40684e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd0cc009960 tx=0x7fd0cc0092d0).ready entity=mon.0 client_cookie=cb9e9add051b3c3c server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d159a700 1 -- 172.21.15.83:0/2657041263 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0cc00b9c0 con 0x7fd0d40680c0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d159a700 1 -- 172.21.15.83:0/2657041263 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd0cc011400 con 0x7fd0d40680c0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.725+0000 7fd0d159a700 1 -- 172.21.15.83:0/2657041263 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0cc010580 con 0x7fd0d40680c0 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 -- 172.21.15.83:0/2657041263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 msgr2=0x7fd0d40684e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:53.765 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 --2- 172.21.15.83:0/2657041263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d40684e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd0cc009960 tx=0x7fd0cc0092d0).stop 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 -- 172.21.15.83:0/2657041263 shutdown_connections 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 --2- 172.21.15.83:0/2657041263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d40684e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 -- 172.21.15.83:0/2657041263 >> 172.21.15.83:0/2657041263 conn(0x7fd0d4063ba0 msgr2=0x7fd0d4065fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 -- 172.21.15.83:0/2657041263 shutdown_connections 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.726+0000 7fd0d359e700 1 -- 172.21.15.83:0/2657041263 wait complete. 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d359e700 1 Processor -- start 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d359e700 1 -- start start 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d359e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d4110ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d359e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d4112820 con 0x7fd0d40680c0 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d4110ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d4110ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53332/0 (socket says 172.21.15.83:53332) 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.727+0000 7fd0d259c700 1 -- 172.21.15.83:0/1127274005 learned_addr learned my addr 172.21.15.83:0/1127274005 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:53.766 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0d259c700 1 -- 172.21.15.83:0/1127274005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc009570 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0d259c700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d4110ca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd0cc011d70 tx=0x7fd0cc011f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0cc0107f0 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd0cc010de0 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0cc01adc0 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d4111190 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.728+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d4111620 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.729+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fd0cc00f070 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.729+0000 7fd0caffd700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.729+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d4045ad0 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.729+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd0cc041b40 con 0x7fd0d40680c0 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.729+0000 7fd0d1d9b700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 msgr2=0x7fd0b40323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/89965517 2024-02-10T09:21:53.767 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.729+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T09:21:53.768 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.733+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0cc018790 con 0x7fd0d40680c0 2024-02-10T09:21:53.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:52.424903+0000 mon.smithi083 (mon.0) 60 : audit [INF] from='client.? 172.21.15.83:0/1632148148' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T09:21:53.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:52.424956+0000 mon.smithi083 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi083.dowdym(active, since 16s) 2024-02-10T09:21:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.930+0000 7fd0d1d9b700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 msgr2=0x7fd0b40323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/89965517 2024-02-10T09:21:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.930+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T09:21:53.931 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:53.931+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 --> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0d4106330 con 0x7fd0b402ff10 2024-02-10T09:21:54.331 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:54.331+0000 7fd0d1d9b700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 msgr2=0x7fd0b40323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/89965517 2024-02-10T09:21:54.332 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:54.331+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-10T09:21:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:55.132+0000 7fd0d1d9b700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 msgr2=0x7fd0b40323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/89965517 2024-02-10T09:21:55.133 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:55.132+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-10T09:21:56.734 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:56.733+0000 7fd0d1d9b700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 msgr2=0x7fd0b40323d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/89965517 2024-02-10T09:21:56.734 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:56.733+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-10T09:21:57.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:57.443+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fd0cc02c910 con 0x7fd0d40680c0 2024-02-10T09:21:57.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:57.443+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 msgr2=0x7fd0b40323d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T09:21:57.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:57.443+0000 7fd0caffd700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:21:57.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:57.389463+0000 mon.smithi083 (mon.0) 62 : cluster [INF] Active manager daemon smithi083.dowdym restarted 2024-02-10T09:21:57.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:57.389843+0000 mon.smithi083 (mon.0) 63 : cluster [INF] Activating manager daemon smithi083.dowdym 2024-02-10T09:21:57.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:57.394069+0000 mon.smithi083 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-10T09:21:58.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.451+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mon.0 v2:172.21.15.83:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fd0cc018d90 con 0x7fd0d40680c0 2024-02-10T09:21:58.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.451+0000 7fd0caffd700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fd0b4033d30 0x7fd0b4036120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:58.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.451+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 --> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0d4106330 con 0x7fd0b4033d30 2024-02-10T09:21:58.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.452+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fd0b4033d30 0x7fd0b4036120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:58.454 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.452+0000 7fd0d1d9b700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fd0b4033d30 0x7fd0b4036120 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd0bc006fd0 tx=0x7fd0bc006db0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:58.457 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.456+0000 7fd0caffd700 1 -- 172.21.15.83:0/1127274005 <== mgr.14142 v2:172.21.15.83:6800/1759090023 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd0d4106330 con 0x7fd0b4033d30 2024-02-10T09:21:58.457 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-10T09:21:58.460 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fd0b4033d30 msgr2=0x7fd0b4036120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:58.460 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fd0b4033d30 0x7fd0b4036120 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd0bc006fd0 tx=0x7fd0bc006db0).stop 2024-02-10T09:21:58.460 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 msgr2=0x7fd0d4110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:58.460 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d4110ca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd0cc011d70 tx=0x7fd0cc011f90).stop 2024-02-10T09:21:58.460 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 shutdown_connections 2024-02-10T09:21:58.461 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0d40680c0 0x7fd0d4110ca0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:58.461 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fd0b4033d30 0x7fd0b4036120 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:58.461 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 --2- 172.21.15.83:0/1127274005 >> [v2:172.21.15.83:6800/89965517,v1:172.21.15.83:6801/89965517] conn(0x7fd0b402ff10 0x7fd0b40323d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:21:58.461 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.458+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 >> 172.21.15.83:0/1127274005 conn(0x7fd0d4063ba0 msgr2=0x7fd0d40664f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:58.461 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.459+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 shutdown_connections 2024-02-10T09:21:58.461 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:58.459+0000 7fd0d359e700 1 -- 172.21.15.83:0/1127274005 wait complete. 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:57.444775+0000 mon.smithi083 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi083.dowdym(active, starting, since 0.0550129s) 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.447315+0000 mon.smithi083 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.447966+0000 mon.smithi083 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mgr metadata", "who": "smithi083.dowdym", "id": "smithi083.dowdym"}]: dispatch 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.448601+0000 mon.smithi083 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.448741+0000 mon.smithi083 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.448866+0000 mon.smithi083 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:57.450756+0000 mon.smithi083 (mon.0) 71 : cluster [INF] Manager daemon smithi083.dowdym is now available 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.547268+0000 mon.smithi083 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:58.763 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.548743+0000 mon.smithi083 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:21:58.764 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.562878+0000 mon.smithi083 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:21:58.764 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:57.565419+0000 mon.smithi083 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:21:59.057 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying grafana service with default placement... 2024-02-10T09:21:59.058 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-10T09:21:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.599+0000 7fbd62c2b700 1 Processor -- start 2024-02-10T09:21:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd62c2b700 1 -- start start 2024-02-10T09:21:59.603 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd62c2b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd62c2b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c0fc880 con 0x7fbd5c0703f0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd609c7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd609c7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53374/0 (socket says 172.21.15.83:53374) 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd609c7700 1 -- 172.21.15.83:0/1747378463 learned_addr learned my addr 172.21.15.83:0/1747378463 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd609c7700 1 -- 172.21.15.83:0/1747378463 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c101380 con 0x7fbd5c0703f0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd609c7700 1 --2- 172.21.15.83:0/1747378463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c100e90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbd5000b5f0 tx=0x7fbd50006ee0).ready entity=mon.0 client_cookie=ca2f8c5946937c9b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd5b7fe700 1 -- 172.21.15.83:0/1747378463 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5000d600 con 0x7fbd5c0703f0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd5b7fe700 1 -- 172.21.15.83:0/1747378463 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbd5000dbf0 con 0x7fbd5c0703f0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.600+0000 7fbd5b7fe700 1 -- 172.21.15.83:0/1747378463 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd50011ce0 con 0x7fbd5c0703f0 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 -- 172.21.15.83:0/1747378463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 msgr2=0x7fbd5c100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 --2- 172.21.15.83:0/1747378463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c100e90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbd5000b5f0 tx=0x7fbd50006ee0).stop 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 -- 172.21.15.83:0/1747378463 shutdown_connections 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 --2- 172.21.15.83:0/1747378463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c100e90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:59.604 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 -- 172.21.15.83:0/1747378463 >> 172.21.15.83:0/1747378463 conn(0x7fbd5c0f6530 msgr2=0x7fbd5c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:59.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 -- 172.21.15.83:0/1747378463 shutdown_connections 2024-02-10T09:21:59.605 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.601+0000 7fbd62c2b700 1 -- 172.21.15.83:0/1747378463 wait complete. 2024-02-10T09:21:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd62c2b700 1 Processor -- start 2024-02-10T09:21:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd62c2b700 1 -- start start 2024-02-10T09:21:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd62c2b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c10cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:59.606 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd62c2b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c10d290 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd609c7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c10cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd609c7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c10cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53378/0 (socket says 172.21.15.83:53378) 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd609c7700 1 -- 172.21.15.83:0/2620184331 learned_addr learned my addr 172.21.15.83:0/2620184331 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd609c7700 1 -- 172.21.15.83:0/2620184331 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5000b200 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.602+0000 7fbd609c7700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c10cda0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbd50011ed0 tx=0x7fbd50006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.603+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5001b570 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.603+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbd5001bb60 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.603+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c10d490 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.603+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd5001aad0 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.603+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c10e0d0 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.603+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fbd5000f070 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.604+0000 7fbd597fa700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fbd4002ff70 0x7fbd40032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.604+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbd500416f0 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.604+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c10d920 con 0x7fbd5c0703f0 2024-02-10T09:21:59.607 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.604+0000 7fbd5bfff700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fbd4002ff70 0x7fbd40032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:21:59.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.604+0000 7fbd5bfff700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fbd4002ff70 0x7fbd40032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbd480096a0 tx=0x7fbd48006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:21:59.608 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.608+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd50016070 con 0x7fbd5c0703f0 2024-02-10T09:21:59.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:58.453825+0000 mon.smithi083 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi083.dowdym(active, since 1.06406s) 2024-02-10T09:21:59.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:58.455069+0000 mgr.smithi083.dowdym (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-10T09:21:59.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:21:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:58.457156+0000 mon.smithi083 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:21:59.785 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.785+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 --> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd5c1021b0 con 0x7fbd4002ff70 2024-02-10T09:21:59.792 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.791+0000 7fbd597fa700 1 -- 172.21.15.83:0/2620184331 <== mgr.14142 v2:172.21.15.83:6800/1759090023 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fbd480092a0 con 0x7fbd4002ff70 2024-02-10T09:21:59.792 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fbd4002ff70 msgr2=0x7fbd40032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fbd4002ff70 0x7fbd40032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbd480096a0 tx=0x7fbd48006b40).stop 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 msgr2=0x7fbd5c10cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c10cda0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbd50011ed0 tx=0x7fbd50006ee0).stop 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 shutdown_connections 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fbd4002ff70 0x7fbd40032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 --2- 172.21.15.83:0/2620184331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd5c0703f0 0x7fbd5c10cda0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 >> 172.21.15.83:0/2620184331 conn(0x7fbd5c0f6530 msgr2=0x7fbd5c0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 shutdown_connections 2024-02-10T09:21:59.795 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:21:59.793+0000 7fbd62c2b700 1 -- 172.21.15.83:0/2620184331 wait complete. 2024-02-10T09:22:00.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:00 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:21:59.458100+0000 mon.smithi083 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi083.dowdym(active, since 2s) 2024-02-10T09:22:00.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:00 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:59.791989+0000 mon.smithi083 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:01.374 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying node-exporter service with default placement... 2024-02-10T09:22:01.374 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-10T09:22:01.638 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:21:59.786552+0000 mgr.smithi083.dowdym (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:01.638 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:21:59.787070+0000 mgr.smithi083.dowdym (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566fd21700 1 Processor -- start 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566fd21700 1 -- start start 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566fd21700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f5668100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566fd21700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56680fc8a0 con 0x7f56680703d0 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566dabd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f5668100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566dabd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f5668100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53380/0 (socket says 172.21.15.83:53380) 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.915+0000 7f566dabd700 1 -- 172.21.15.83:0/296353671 learned_addr learned my addr 172.21.15.83:0/296353671 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566dabd700 1 -- 172.21.15.83:0/296353671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56681013a0 con 0x7f56680703d0 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566dabd700 1 --2- 172.21.15.83:0/296353671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f5668100eb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f566400b5f0 tx=0x7f5664006ee0).ready entity=mon.0 client_cookie=e16b9b33b989f0bf server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566cabb700 1 -- 172.21.15.83:0/296353671 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566400d600 con 0x7f56680703d0 2024-02-10T09:22:01.918 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566cabb700 1 -- 172.21.15.83:0/296353671 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f566400dbf0 con 0x7f56680703d0 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566cabb700 1 -- 172.21.15.83:0/296353671 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5664011cb0 con 0x7f56680703d0 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566fd21700 1 -- 172.21.15.83:0/296353671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 msgr2=0x7f5668100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566fd21700 1 --2- 172.21.15.83:0/296353671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f5668100eb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f566400b5f0 tx=0x7f5664006ee0).stop 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566fd21700 1 -- 172.21.15.83:0/296353671 shutdown_connections 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566fd21700 1 --2- 172.21.15.83:0/296353671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f5668100eb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.916+0000 7f566fd21700 1 -- 172.21.15.83:0/296353671 >> 172.21.15.83:0/296353671 conn(0x7f56680f6530 msgr2=0x7f56680f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.917+0000 7f566fd21700 1 -- 172.21.15.83:0/296353671 shutdown_connections 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.917+0000 7f566fd21700 1 -- 172.21.15.83:0/296353671 wait complete. 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.917+0000 7f566fd21700 1 Processor -- start 2024-02-10T09:22:01.919 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.917+0000 7f566fd21700 1 -- start start 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566fd21700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f566810ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566fd21700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566810d310 con 0x7f56680703d0 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566dabd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f566810ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566dabd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f566810ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53394/0 (socket says 172.21.15.83:53394) 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566dabd700 1 -- 172.21.15.83:0/3180769824 learned_addr learned my addr 172.21.15.83:0/3180769824 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566dabd700 1 -- 172.21.15.83:0/3180769824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566400b200 con 0x7f56680703d0 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566dabd700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f566810ce20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5664006c70 tx=0x7f5664006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5664011400 con 0x7f56680703d0 2024-02-10T09:22:01.921 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566810d510 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f566401b8a0 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566401a7c0 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.918+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566810e120 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.919+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f566400f070 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.919+0000 7f56567fc700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f565002ff70 0x7f5650032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.919+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f566401be40 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.919+0000 7f566d2bc700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f565002ff70 0x7f5650032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.919+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5668059100 con 0x7f56680703d0 2024-02-10T09:22:01.922 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.920+0000 7f566d2bc700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f565002ff70 0x7f5650032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f56580096a0 tx=0x7f5658006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:01.924 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:01.923+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5664016030 con 0x7f56680703d0 2024-02-10T09:22:02.098 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.098+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 --> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f56681020f0 con 0x7f565002ff70 2024-02-10T09:22:02.102 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.102+0000 7f56567fc700 1 -- 172.21.15.83:0/3180769824 <== mgr.14142 v2:172.21.15.83:6800/1759090023 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f56681020f0 con 0x7f565002ff70 2024-02-10T09:22:02.102 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-10T09:22:02.105 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f565002ff70 msgr2=0x7f5650032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:02.105 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f565002ff70 0x7f5650032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f56580096a0 tx=0x7f5658006b40).stop 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 msgr2=0x7f566810ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f566810ce20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f5664006c70 tx=0x7f5664006ee0).stop 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 shutdown_connections 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f565002ff70 0x7f5650032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 --2- 172.21.15.83:0/3180769824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56680703d0 0x7f566810ce20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.103+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 >> 172.21.15.83:0/3180769824 conn(0x7f56680f6530 msgr2=0x7f56680f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.104+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 shutdown_connections 2024-02-10T09:22:02.106 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:02.104+0000 7f566fd21700 1 -- 172.21.15.83:0/3180769824 wait complete. 2024-02-10T09:22:02.730 INFO:teuthology.orchestra.run.smithi083.stderr:Deploying alertmanager service with default placement... 2024-02-10T09:22:02.730 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.287+0000 7fb1167c2700 1 Processor -- start 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.287+0000 7fb1167c2700 1 -- start start 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.287+0000 7fb1167c2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.287+0000 7fb1167c2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100fc7b0 con 0x7fb1100fbea0 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53410/0 (socket says 172.21.15.83:53410) 2024-02-10T09:22:03.291 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10ffff700 1 -- 172.21.15.83:0/87107537 learned_addr learned my addr 172.21.15.83:0/87107537 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10ffff700 1 -- 172.21.15.83:0/87107537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1100fc8f0 con 0x7fb1100fbea0 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10ffff700 1 --2- 172.21.15.83:0/87107537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100fc2c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb0f800b5f0 tx=0x7fb0f8006ee0).ready entity=mon.0 client_cookie=6a4bf1b516a53299 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10effd700 1 -- 172.21.15.83:0/87107537 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f800d600 con 0x7fb1100fbea0 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10effd700 1 -- 172.21.15.83:0/87107537 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb0f800dbf0 con 0x7fb1100fbea0 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.288+0000 7fb10effd700 1 -- 172.21.15.83:0/87107537 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f8011cb0 con 0x7fb1100fbea0 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 -- 172.21.15.83:0/87107537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 msgr2=0x7fb1100fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 --2- 172.21.15.83:0/87107537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100fc2c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb0f800b5f0 tx=0x7fb0f8006ee0).stop 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 -- 172.21.15.83:0/87107537 shutdown_connections 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 --2- 172.21.15.83:0/87107537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100fc2c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 -- 172.21.15.83:0/87107537 >> 172.21.15.83:0/87107537 conn(0x7fb1100f76a0 msgr2=0x7fb1100f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 -- 172.21.15.83:0/87107537 shutdown_connections 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 -- 172.21.15.83:0/87107537 wait complete. 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.289+0000 7fb1167c2700 1 Processor -- start 2024-02-10T09:22:03.292 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb1167c2700 1 -- start start 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb1167c2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb1167c2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb110070a60 con 0x7fb1100fbea0 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:53416/0 (socket says 172.21.15.83:53416) 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10ffff700 1 -- 172.21.15.83:0/2205184063 learned_addr learned my addr 172.21.15.83:0/2205184063 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10ffff700 1 -- 172.21.15.83:0/2205184063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f800b200 con 0x7fb1100fbea0 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10ffff700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100725e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fb0f800bf40 tx=0x7fb0f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:03.293 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f801b570 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb110070c00 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb0f801bb60 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0f801aa20 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.290+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb110071090 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.291+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb0f800f070 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.291+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb110045ad0 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.291+0000 7fb10cff9700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fb0f002ff20 0x7fb0f00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.291+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb0f8041820 con 0x7fb1100fbea0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.291+0000 7fb10f7fe700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fb0f002ff20 0x7fb0f00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:03.294 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.292+0000 7fb10f7fe700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fb0f002ff20 0x7fb0f00323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb1000096a0 tx=0x7fb100006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:03.296 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.295+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0f8010360 con 0x7fb1100fbea0 2024-02-10T09:22:03.473 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.472+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 --> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1100fdd80 con 0x7fb0f002ff20 2024-02-10T09:22:03.476 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.476+0000 7fb10cff9700 1 -- 172.21.15.83:0/2205184063 <== mgr.14142 v2:172.21.15.83:6800/1759090023 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb1100fdd80 con 0x7fb0f002ff20 2024-02-10T09:22:03.476 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fb0f002ff20 msgr2=0x7fb0f00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fb0f002ff20 0x7fb0f00323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb1000096a0 tx=0x7fb100006b40).stop 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 msgr2=0x7fb1100725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100725e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fb0f800bf40 tx=0x7fb0f8006ee0).stop 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 shutdown_connections 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fb0f002ff20 0x7fb0f00323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 --2- 172.21.15.83:0/2205184063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1100fbea0 0x7fb1100725e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 >> 172.21.15.83:0/2205184063 conn(0x7fb1100f76a0 msgr2=0x7fb1100f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 shutdown_connections 2024-02-10T09:22:03.479 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:03.478+0000 7fb1167c2700 1 -- 172.21.15.83:0/2205184063 wait complete. 2024-02-10T09:22:03.506 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:02.099336+0000 mgr.smithi083.dowdym (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:03.506 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:22:02.100053+0000 mgr.smithi083.dowdym (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-10T09:22:03.506 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:02.102261+0000 mon.smithi083 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:04.088 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-10T09:22:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.674+0000 7f89128d5700 1 Processor -- start 2024-02-10T09:22:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89128d5700 1 -- start start 2024-02-10T09:22:04.715 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89128d5700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c0faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89128d5700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c0fb440 con 0x7f890c0fab30 2024-02-10T09:22:04.719 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89118d3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c0faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89118d3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c0faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40854/0 (socket says 172.21.15.83:40854) 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89118d3700 1 -- 172.21.15.83:0/2260068386 learned_addr learned my addr 172.21.15.83:0/2260068386 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89118d3700 1 -- 172.21.15.83:0/2260068386 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c0fb580 con 0x7f890c0fab30 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89118d3700 1 --2- 172.21.15.83:0/2260068386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c0faf50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f88fc00b5f0 tx=0x7f88fc006ee0).ready entity=mon.0 client_cookie=16bcec40bf81e6c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89108d1700 1 -- 172.21.15.83:0/2260068386 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88fc00d600 con 0x7f890c0fab30 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89108d1700 1 -- 172.21.15.83:0/2260068386 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f88fc00dbf0 con 0x7f890c0fab30 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.675+0000 7f89108d1700 1 -- 172.21.15.83:0/2260068386 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88fc011cb0 con 0x7f890c0fab30 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 -- 172.21.15.83:0/2260068386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 msgr2=0x7f890c0faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 --2- 172.21.15.83:0/2260068386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c0faf50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f88fc00b5f0 tx=0x7f88fc006ee0).stop 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 -- 172.21.15.83:0/2260068386 shutdown_connections 2024-02-10T09:22:04.720 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 --2- 172.21.15.83:0/2260068386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c0faf50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 -- 172.21.15.83:0/2260068386 >> 172.21.15.83:0/2260068386 conn(0x7f890c0f6390 msgr2=0x7f890c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 -- 172.21.15.83:0/2260068386 shutdown_connections 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 -- 172.21.15.83:0/2260068386 wait complete. 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 Processor -- start 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.676+0000 7f89128d5700 1 -- start start 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89128d5700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c10d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89128d5700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c070450 con 0x7f890c0fab30 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89118d3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c10d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89118d3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c10d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40868/0 (socket says 172.21.15.83:40868) 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89118d3700 1 -- 172.21.15.83:0/972180434 learned_addr learned my addr 172.21.15.83:0/972180434 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89118d3700 1 -- 172.21.15.83:0/972180434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc00b200 con 0x7f890c0fab30 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89118d3700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c10d700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f88fc002f40 tx=0x7f88fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:04.721 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88fc01b410 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f88fc01ba00 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88fc01a900 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c070650 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.677+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c10ddb0 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.678+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f88fc00f070 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.678+0000 7f88fa7fc700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f88f402ff70 0x7f88f4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.678+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f88fc041470 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.678+0000 7f89110d2700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f88f402ff70 0x7f88f4032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.678+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c059100 con 0x7f890c0fab30 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.679+0000 7f89110d2700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f88f402ff70 0x7f88f4032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:04.722 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.682+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88fc016070 con 0x7f890c0fab30 2024-02-10T09:22:04.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:03.473834+0000 mgr.smithi083.dowdym (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:04.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:22:03.474326+0000 mgr.smithi083.dowdym (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-10T09:22:04.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:03.476426+0000 mon.smithi083 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:04.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.836+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f890c0fc9d0 con 0x7f890c0fab30 2024-02-10T09:22:04.839 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.839+0000 7f88fa7fc700 1 -- 172.21.15.83:0/972180434 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f88fc010360 con 0x7f890c0fab30 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f88f402ff70 msgr2=0x7f88f4032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f88f402ff70 0x7f88f4032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f89080096a0 tx=0x7f8908006b40).stop 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 msgr2=0x7f890c10d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c10d700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f88fc002f40 tx=0x7f88fc006ee0).stop 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 shutdown_connections 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f88f402ff70 0x7f88f4032430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 --2- 172.21.15.83:0/972180434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f890c0fab30 0x7f890c10d700 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.840+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 >> 172.21.15.83:0/972180434 conn(0x7f890c0f6390 msgr2=0x7f890c0f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.841+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 shutdown_connections 2024-02-10T09:22:04.842 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:04.841+0000 7f89128d5700 1 -- 172.21.15.83:0/972180434 wait complete. 2024-02-10T09:22:05.548 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-10T09:22:06.220 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.215+0000 7f57b4d96700 1 Processor -- start 2024-02-10T09:22:06.220 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.215+0000 7f57b4d96700 1 -- start start 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.215+0000 7f57b4d96700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b00fb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.215+0000 7f57b4d96700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b00fb700 con 0x7f57b00fadf0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ae59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b00fb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ae59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b00fb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40882/0 (socket says 172.21.15.83:40882) 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ae59c700 1 -- 172.21.15.83:0/2400028485 learned_addr learned my addr 172.21.15.83:0/2400028485 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ae59c700 1 -- 172.21.15.83:0/2400028485 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b00fb840 con 0x7f57b00fadf0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ae59c700 1 --2- 172.21.15.83:0/2400028485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b00fb210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f57a400b5f0 tx=0x7f57a4006ee0).ready entity=mon.0 client_cookie=3e427031fe9ec79d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ad59a700 1 -- 172.21.15.83:0/2400028485 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a400d600 con 0x7f57b00fadf0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ad59a700 1 -- 172.21.15.83:0/2400028485 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f57a400dbf0 con 0x7f57b00fadf0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.216+0000 7f57ad59a700 1 -- 172.21.15.83:0/2400028485 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a4011cb0 con 0x7f57b00fadf0 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 -- 172.21.15.83:0/2400028485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 msgr2=0x7f57b00fb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 --2- 172.21.15.83:0/2400028485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b00fb210 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f57a400b5f0 tx=0x7f57a4006ee0).stop 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 -- 172.21.15.83:0/2400028485 shutdown_connections 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 --2- 172.21.15.83:0/2400028485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b00fb210 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:06.221 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 -- 172.21.15.83:0/2400028485 >> 172.21.15.83:0/2400028485 conn(0x7f57b00f65c0 msgr2=0x7f57b00f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 -- 172.21.15.83:0/2400028485 shutdown_connections 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 -- 172.21.15.83:0/2400028485 wait complete. 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 Processor -- start 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.217+0000 7f57b4d96700 1 -- start start 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57b4d96700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b010d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57b4d96700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b00704e0 con 0x7f57b00fadf0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57ae59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b010d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57ae59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b010d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40888/0 (socket says 172.21.15.83:40888) 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57ae59c700 1 -- 172.21.15.83:0/2377710991 learned_addr learned my addr 172.21.15.83:0/2377710991 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57ae59c700 1 -- 172.21.15.83:0/2377710991 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a400b200 con 0x7f57b00fadf0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57ae59c700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b010d8c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f57a400dfc0 tx=0x7f57a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a401b410 con 0x7f57b00fadf0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f57a401ba00 con 0x7f57b00fadf0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a401a990 con 0x7f57b00fadf0 2024-02-10T09:22:06.222 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b00706e0 con 0x7f57b00fadf0 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.218+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b010df70 con 0x7f57b00fadf0 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.219+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f57a400f070 con 0x7f57b00fadf0 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.219+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b0045ad0 con 0x7f57b00fadf0 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.219+0000 7f579effd700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f579402ff70 0x7f5794032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.219+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f57a4041450 con 0x7f57b00fadf0 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.220+0000 7f57add9b700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f579402ff70 0x7f5794032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:06.223 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.220+0000 7f57add9b700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f579402ff70 0x7f5794032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f57a00096a0 tx=0x7f57a0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:06.224 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.224+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57a4019af0 con 0x7f57b00fadf0 2024-02-10T09:22:06.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:04.839515+0000 mon.smithi083 (mon.0) 82 : audit [INF] from='client.? 172.21.15.83:0/972180434' entity='client.admin' 2024-02-10T09:22:06.381 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.380+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f57b00fcb00 con 0x7f57b00fadf0 2024-02-10T09:22:06.384 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.383+0000 7f579effd700 1 -- 172.21.15.83:0/2377710991 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f57a4016020 con 0x7f57b00fadf0 2024-02-10T09:22:06.386 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f579402ff70 msgr2=0x7f5794032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:06.386 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f579402ff70 0x7f5794032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f57a00096a0 tx=0x7f57a0006b40).stop 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 msgr2=0x7f57b010d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b010d8c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f57a400dfc0 tx=0x7f57a4006ee0).stop 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 shutdown_connections 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f579402ff70 0x7f5794032430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 --2- 172.21.15.83:0/2377710991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57b00fadf0 0x7f57b010d8c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.385+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 >> 172.21.15.83:0/2377710991 conn(0x7f57b00f65c0 msgr2=0x7f57b00f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.386+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 shutdown_connections 2024-02-10T09:22:06.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:06.386+0000 7f57b4d96700 1 -- 172.21.15.83:0/2377710991 wait complete. 2024-02-10T09:22:07.259 INFO:teuthology.orchestra.run.smithi083.stderr:Enabling the dashboard module... 2024-02-10T09:22:07.259 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-10T09:22:07.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:06.384078+0000 mon.smithi083 (mon.0) 83 : audit [INF] from='client.? 172.21.15.83:0/2377710991' entity='client.admin' 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed41126700 1 Processor -- start 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed41126700 1 -- start start 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed41126700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed41126700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c072980 con 0x7fed3c072560 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed3ad9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed3ad9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40890/0 (socket says 172.21.15.83:40890) 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed3ad9d700 1 -- 172.21.15.83:0/1884879814 learned_addr learned my addr 172.21.15.83:0/1884879814 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.271+0000 7fed3ad9d700 1 -- 172.21.15.83:0/1884879814 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3c070ff0 con 0x7fed3c072560 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.272+0000 7fed3ad9d700 1 --2- 172.21.15.83:0/1884879814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c070b00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fed2c00b5f0 tx=0x7fed2c006ee0).ready entity=mon.0 client_cookie=4c9a43bc95425f29 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:08.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.272+0000 7fed39d9b700 1 -- 172.21.15.83:0/1884879814 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed2c00d600 con 0x7fed3c072560 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.272+0000 7fed39d9b700 1 -- 172.21.15.83:0/1884879814 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed2c00dc30 con 0x7fed3c072560 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.272+0000 7fed39d9b700 1 -- 172.21.15.83:0/1884879814 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed2c011d30 con 0x7fed3c072560 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- 172.21.15.83:0/1884879814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 msgr2=0x7fed3c070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 --2- 172.21.15.83:0/1884879814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c070b00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fed2c00b5f0 tx=0x7fed2c006ee0).stop 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- 172.21.15.83:0/1884879814 shutdown_connections 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 --2- 172.21.15.83:0/1884879814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c070b00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:08.276 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- 172.21.15.83:0/1884879814 >> 172.21.15.83:0/1884879814 conn(0x7fed3c0f76a0 msgr2=0x7fed3c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- 172.21.15.83:0/1884879814 shutdown_connections 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- 172.21.15.83:0/1884879814 wait complete. 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 Processor -- start 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- start start 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.273+0000 7fed41126700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3c10d3b0 con 0x7fed3c072560 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed3ad9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed3ad9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40896/0 (socket says 172.21.15.83:40896) 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed3ad9d700 1 -- 172.21.15.83:0/3484834177 learned_addr learned my addr 172.21.15.83:0/3484834177 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed3ad9d700 1 -- 172.21.15.83:0/3484834177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2c00b200 con 0x7fed3c072560 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed3ad9d700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c10cec0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fed2c00e040 tx=0x7fed2c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed2c00d9e0 con 0x7fed3c072560 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed2c01b8e0 con 0x7fed3c072560 2024-02-10T09:22:08.277 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed3c10d5b0 con 0x7fed3c072560 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed2c01a820 con 0x7fed3c072560 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.274+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c10e160 con 0x7fed3c072560 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.275+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fed2c00f070 con 0x7fed3c072560 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.275+0000 7fed237fe700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fed2802ff20 0x7fed280323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.275+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3c059100 con 0x7fed3c072560 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.275+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fed2c041510 con 0x7fed3c072560 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.275+0000 7fed3a59c700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fed2802ff20 0x7fed280323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:08.278 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.276+0000 7fed3a59c700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fed2802ff20 0x7fed280323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fed300096a0 tx=0x7fed30006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:08.281 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.280+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed2c025460 con 0x7fed3c072560 2024-02-10T09:22:08.485 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.484+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fed3c106910 con 0x7fed3c072560 2024-02-10T09:22:08.964 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.963+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fed2c0257e0 con 0x7fed3c072560 2024-02-10T09:22:08.965 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.964+0000 7fed237fe700 1 -- 172.21.15.83:0/3484834177 <== mon.0 v2:172.21.15.83:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fed2c040c70 con 0x7fed3c072560 2024-02-10T09:22:08.967 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.966+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fed2802ff20 msgr2=0x7fed280323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.966+0000 7fed41126700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fed2802ff20 0x7fed280323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fed300096a0 tx=0x7fed30006b40).stop 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 msgr2=0x7fed3c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c10cec0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fed2c00e040 tx=0x7fed2c006ee0).stop 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 shutdown_connections 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fed2802ff20 0x7fed280323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 --2- 172.21.15.83:0/3484834177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c072560 0x7fed3c10cec0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 >> 172.21.15.83:0/3484834177 conn(0x7fed3c0f76a0 msgr2=0x7fed3c0f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 shutdown_connections 2024-02-10T09:22:08.968 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:08.967+0000 7fed41126700 1 -- 172.21.15.83:0/3484834177 wait complete. 2024-02-10T09:22:08.994 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:07.704643+0000 mon.smithi083 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:08.994 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:07.963722+0000 mon.smithi083 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:08.994 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:08.485304+0000 mon.smithi083 (mon.0) 86 : audit [INF] from='client.? 172.21.15.83:0/3484834177' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T09:22:09.616 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-10T09:22:10.175 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:08.937804+0000 mon.smithi083 (mon.0) 87 : audit [INF] from='mgr.14142 172.21.15.83:0/415022633' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:10.175 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:08.965416+0000 mon.smithi083 (mon.0) 88 : audit [INF] from='client.? 172.21.15.83:0/3484834177' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T09:22:10.175 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:08.965462+0000 mon.smithi083 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi083.dowdym(active, since 11s) 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.198+0000 7fc4e434b700 1 Processor -- start 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.198+0000 7fc4e434b700 1 -- start start 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.199+0000 7fc4e434b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc0fc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.199+0000 7fc4e434b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0fcd70 con 0x7fc4dc070430 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.199+0000 7fc4e20e7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc0fc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.199+0000 7fc4e20e7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc0fc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40920/0 (socket says 172.21.15.83:40920) 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.199+0000 7fc4e20e7700 1 -- 172.21.15.83:0/3245582821 learned_addr learned my addr 172.21.15.83:0/3245582821 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.200+0000 7fc4e20e7700 1 -- 172.21.15.83:0/3245582821 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0fceb0 con 0x7fc4dc070430 2024-02-10T09:22:10.272 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.200+0000 7fc4e20e7700 1 --2- 172.21.15.83:0/3245582821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc0fc880 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc4d800b5f0 tx=0x7fc4d8006ee0).ready entity=mon.0 client_cookie=151dc655d719ae2c server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.200+0000 7fc4e10e5700 1 -- 172.21.15.83:0/3245582821 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d800d600 con 0x7fc4dc070430 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.200+0000 7fc4e10e5700 1 -- 172.21.15.83:0/3245582821 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4d800dc30 con 0x7fc4dc070430 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.200+0000 7fc4e10e5700 1 -- 172.21.15.83:0/3245582821 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d8011d30 con 0x7fc4dc070430 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 -- 172.21.15.83:0/3245582821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 msgr2=0x7fc4dc0fc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 --2- 172.21.15.83:0/3245582821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc0fc880 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fc4d800b5f0 tx=0x7fc4d8006ee0).stop 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 -- 172.21.15.83:0/3245582821 shutdown_connections 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 --2- 172.21.15.83:0/3245582821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc0fc880 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 -- 172.21.15.83:0/3245582821 >> 172.21.15.83:0/3245582821 conn(0x7fc4dc0f6550 msgr2=0x7fc4dc0f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 -- 172.21.15.83:0/3245582821 shutdown_connections 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 -- 172.21.15.83:0/3245582821 wait complete. 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 Processor -- start 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 -- start start 2024-02-10T09:22:10.273 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.201+0000 7fc4e434b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4e434b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc111680 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4e20e7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4e20e7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40932/0 (socket says 172.21.15.83:40932) 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4e20e7700 1 -- 172.21.15.83:0/1800214121 learned_addr learned my addr 172.21.15.83:0/1800214121 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4e20e7700 1 -- 172.21.15.83:0/1800214121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d800b200 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4e20e7700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc111190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc4d800e040 tx=0x7fc4d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d800d9e0 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4d801b8e0 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.202+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4d801a820 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.203+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc111880 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.203+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc111d10 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.204+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc045ad0 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.204+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc4d800f070 con 0x7fc4dc070430 2024-02-10T09:22:10.274 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.204+0000 7fc4caffd700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 0x7fc4c4032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.204+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc4d8040880 con 0x7fc4dc070430 2024-02-10T09:22:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.204+0000 7fc4e18e6700 1 -- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 msgr2=0x7fc4c4032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/1759090023 2024-02-10T09:22:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.204+0000 7fc4e18e6700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 0x7fc4c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T09:22:10.275 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.212+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4d8019300 con 0x7fc4dc070430 2024-02-10T09:22:10.405 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.405+0000 7fc4e18e6700 1 -- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 msgr2=0x7fc4c4032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/1759090023 2024-02-10T09:22:10.406 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.405+0000 7fc4e18e6700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 0x7fc4c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T09:22:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.450+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc4dc112e70 con 0x7fc4dc070430 2024-02-10T09:22:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.450+0000 7fc4caffd700 1 -- 172.21.15.83:0/1800214121 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fc4d8016030 con 0x7fc4dc070430 2024-02-10T09:22:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:22:10.452 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-10T09:22:10.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "available": true, 2024-02-10T09:22:10.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "active_name": "smithi083.dowdym", 2024-02-10T09:22:10.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T09:22:10.453 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:22:10.454 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 msgr2=0x7fc4c4032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 0x7fc4c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 msgr2=0x7fc4dc111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc111190 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fc4d800e040 tx=0x7fc4d8006ee0).stop 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 shutdown_connections 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7fc4c402ffc0 0x7fc4c4032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 --2- 172.21.15.83:0/1800214121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4dc070430 0x7fc4dc111190 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 >> 172.21.15.83:0/1800214121 conn(0x7fc4dc0f6550 msgr2=0x7fc4dc0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 shutdown_connections 2024-02-10T09:22:10.455 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:10.452+0000 7fc4e434b700 1 -- 172.21.15.83:0/1800214121 wait complete. 2024-02-10T09:22:11.230 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for the mgr to restart... 2024-02-10T09:22:11.231 INFO:teuthology.orchestra.run.smithi083.stderr:Waiting for mgr epoch 13... 2024-02-10T09:22:11.231 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-10T09:22:11.233 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:10.451391+0000 mon.smithi083 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.83:0/1800214121' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e7557700 1 Processor -- start 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e7557700 1 -- start start 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e7557700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e0100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e7557700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00fc880 con 0x7f26e00703f0 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e52f3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e0100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e52f3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e0100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40936/0 (socket says 172.21.15.83:40936) 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.779+0000 7f26e52f3700 1 -- 172.21.15.83:0/4196607173 learned_addr learned my addr 172.21.15.83:0/4196607173 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.780+0000 7f26e52f3700 1 -- 172.21.15.83:0/4196607173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0101380 con 0x7f26e00703f0 2024-02-10T09:22:11.781 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.780+0000 7f26e52f3700 1 --2- 172.21.15.83:0/4196607173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e0100e90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f26dc00b5f0 tx=0x7f26dc006ee0).ready entity=mon.0 client_cookie=64fd55d25426f489 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.780+0000 7f26d7fff700 1 -- 172.21.15.83:0/4196607173 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26dc00d600 con 0x7f26e00703f0 2024-02-10T09:22:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.780+0000 7f26d7fff700 1 -- 172.21.15.83:0/4196607173 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26dc00dc30 con 0x7f26e00703f0 2024-02-10T09:22:11.782 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.780+0000 7f26d7fff700 1 -- 172.21.15.83:0/4196607173 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26dc011d60 con 0x7f26e00703f0 2024-02-10T09:22:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.781+0000 7f26e7557700 1 -- 172.21.15.83:0/4196607173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 msgr2=0x7f26e0100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.781+0000 7f26e7557700 1 --2- 172.21.15.83:0/4196607173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e0100e90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f26dc00b5f0 tx=0x7f26dc006ee0).stop 2024-02-10T09:22:11.783 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.781+0000 7f26e7557700 1 -- 172.21.15.83:0/4196607173 shutdown_connections 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.781+0000 7f26e7557700 1 --2- 172.21.15.83:0/4196607173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e0100e90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.781+0000 7f26e7557700 1 -- 172.21.15.83:0/4196607173 >> 172.21.15.83:0/4196607173 conn(0x7f26e00f6530 msgr2=0x7f26e00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.781+0000 7f26e7557700 1 -- 172.21.15.83:0/4196607173 shutdown_connections 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.782+0000 7f26e7557700 1 -- 172.21.15.83:0/4196607173 wait complete. 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.782+0000 7f26e7557700 1 Processor -- start 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.782+0000 7f26e7557700 1 -- start start 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.782+0000 7f26e7557700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e010cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.782+0000 7f26e7557700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e010d2b0 con 0x7f26e00703f0 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e52f3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e010cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e52f3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e010cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:40952/0 (socket says 172.21.15.83:40952) 2024-02-10T09:22:11.784 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e52f3700 1 -- 172.21.15.83:0/1720489621 learned_addr learned my addr 172.21.15.83:0/1720489621 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e52f3700 1 -- 172.21.15.83:0/1720489621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26dc00b200 con 0x7f26e00703f0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e52f3700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e010cdc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f26dc006b80 tx=0x7f26dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26dc011e10 con 0x7f26e00703f0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26dc011610 con 0x7f26e00703f0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e010d4b0 con 0x7f26e00703f0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26dc01a7b0 con 0x7f26e00703f0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.783+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e010e0f0 con 0x7f26e00703f0 2024-02-10T09:22:11.786 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.784+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f26dc00f070 con 0x7f26e00703f0 2024-02-10T09:22:11.787 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.784+0000 7f26d5ffb700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:11.787 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.784+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f26dc041f40 con 0x7f26e00703f0 2024-02-10T09:22:11.787 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.784+0000 7f26e4af2700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 msgr2=0x7f26c4032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/1759090023 2024-02-10T09:22:11.787 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.784+0000 7f26e4af2700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T09:22:11.789 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.789+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 --> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26e0059100 con 0x7f26c402ffc0 2024-02-10T09:22:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.985+0000 7f26e4af2700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 msgr2=0x7f26c4032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/1759090023 2024-02-10T09:22:11.986 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:11.985+0000 7f26e4af2700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T09:22:12.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:12.386+0000 7f26e4af2700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 msgr2=0x7f26c4032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/1759090023 2024-02-10T09:22:12.387 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:12.386+0000 7f26e4af2700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-10T09:22:13.187 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:13.187+0000 7f26e4af2700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 msgr2=0x7f26c4032480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.83:6800/1759090023 2024-02-10T09:22:13.188 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:13.187+0000 7f26e4af2700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-10T09:22:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:14.029+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f26dc024500 con 0x7f26e00703f0 2024-02-10T09:22:14.031 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:14.030+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 msgr2=0x7f26c4032480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T09:22:14.032 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:14.030+0000 7f26d5ffb700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:22:14.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:13.977610+0000 mon.smithi083 (mon.0) 91 : cluster [INF] Active manager daemon smithi083.dowdym restarted 2024-02-10T09:22:14.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:13.977983+0000 mon.smithi083 (mon.0) 92 : cluster [INF] Activating manager daemon smithi083.dowdym 2024-02-10T09:22:14.260 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:13.980197+0000 mon.smithi083 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T09:22:15.036 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.035+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f26dc01a910 con 0x7f26e00703f0 2024-02-10T09:22:15.037 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.035+0000 7f26d5ffb700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f26c4033aa0 0x7f26c4035e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:15.037 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.035+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26e0059100 con 0x7f26c4033aa0 2024-02-10T09:22:15.037 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.035+0000 7f26e4af2700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f26c4033aa0 0x7f26c4035e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:15.037 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.035+0000 7f26e4af2700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f26c4033aa0 0x7f26c4035e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f26d00036d0 tx=0x7f26d0006d20).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:15.037 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.036+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mgr.14162 v2:172.21.15.83:6800/1369708861 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f26e0059100 con 0x7f26c4033aa0 2024-02-10T09:22:15.043 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.042+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f26e010d940 con 0x7f26c4033aa0 2024-02-10T09:22:15.043 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.042+0000 7f26d5ffb700 1 -- 172.21.15.83:0/1720489621 <== mgr.14162 v2:172.21.15.83:6800/1369708861 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f26e010d940 con 0x7f26c4033aa0 2024-02-10T09:22:15.043 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: { 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: } 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f26c4033aa0 msgr2=0x7f26c4035e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f26c4033aa0 0x7f26c4035e90 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f26d00036d0 tx=0x7f26d0006d20).stop 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 msgr2=0x7f26e010cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:15.044 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e010cdc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f26dc006b80 tx=0x7f26dc006ee0).stop 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 shutdown_connections 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f26c4033aa0 0x7f26c4035e90 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26e00703f0 0x7f26e010cdc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 --2- 172.21.15.83:0/1720489621 >> [v2:172.21.15.83:6800/1759090023,v1:172.21.15.83:6801/1759090023] conn(0x7f26c402ffc0 0x7f26c4032480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.043+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 >> 172.21.15.83:0/1720489621 conn(0x7f26e00f6530 msgr2=0x7f26e00f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.044+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 shutdown_connections 2024-02-10T09:22:15.045 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:15.044+0000 7f26e7557700 1 -- 172.21.15.83:0/1720489621 wait complete. 2024-02-10T09:22:15.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:14.031847+0000 mon.smithi083 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi083.dowdym(active, starting, since 0.0539442s) 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.032835+0000 mon.smithi083 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata", "id": "smithi083"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.034840+0000 mon.smithi083 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mgr metadata", "who": "smithi083.dowdym", "id": "smithi083.dowdym"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.034998+0000 mon.smithi083 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.035114+0000 mon.smithi083 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.035227+0000 mon.smithi083 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:14.048954+0000 mon.smithi083 (mon.0) 100 : cluster [INF] Manager daemon smithi083.dowdym is now available 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.146488+0000 mon.smithi083 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.148604+0000 mon.smithi083 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.163989+0000 mon.smithi083 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:22:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:14.172827+0000 mon.smithi083 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:22:15.948 INFO:teuthology.orchestra.run.smithi083.stderr:mgr epoch 13 is available 2024-02-10T09:22:15.948 INFO:teuthology.orchestra.run.smithi083.stderr:Generating a dashboard self-signed certificate... 2024-02-10T09:22:15.948 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-10T09:22:16.224 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:16 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:15.036687+0000 mgr.smithi083.dowdym (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T09:22:16.225 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:16 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:15.037455+0000 mon.smithi083 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi083.dowdym(active, since 1.05956s) 2024-02-10T09:22:16.225 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:16 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:15.043373+0000 mgr.smithi083.dowdym (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T09:22:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f333a4c6700 1 Processor -- start 2024-02-10T09:22:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f333a4c6700 1 -- start start 2024-02-10T09:22:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f333a4c6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f33340fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f333a4c6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340fef80 con 0x7f33340703f0 2024-02-10T09:22:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f3333fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f33340fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:16.629 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f3333fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f33340fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44594/0 (socket says 172.21.15.83:44594) 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f3333fff700 1 -- 172.21.15.83:0/465179796 learned_addr learned my addr 172.21.15.83:0/465179796 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f3333fff700 1 -- 172.21.15.83:0/465179796 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340ff0c0 con 0x7f33340703f0 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.595+0000 7f3333fff700 1 --2- 172.21.15.83:0/465179796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f33340fea90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=1b11ae5c5f5619a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f3332ffd700 1 -- 172.21.15.83:0/465179796 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332400d600 con 0x7f33340703f0 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f3332ffd700 1 -- 172.21.15.83:0/465179796 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f332400dc30 con 0x7f33340703f0 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f3332ffd700 1 -- 172.21.15.83:0/465179796 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324011d60 con 0x7f33340703f0 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 -- 172.21.15.83:0/465179796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 msgr2=0x7f33340fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 --2- 172.21.15.83:0/465179796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f33340fea90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).stop 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 -- 172.21.15.83:0/465179796 shutdown_connections 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 --2- 172.21.15.83:0/465179796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f33340fea90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 -- 172.21.15.83:0/465179796 >> 172.21.15.83:0/465179796 conn(0x7f33340f6530 msgr2=0x7f33340f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 -- 172.21.15.83:0/465179796 shutdown_connections 2024-02-10T09:22:16.630 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.596+0000 7f333a4c6700 1 -- 172.21.15.83:0/465179796 wait complete. 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f333a4c6700 1 Processor -- start 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f333a4c6700 1 -- start start 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f333a4c6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f333410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f333a4c6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333410b060 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f3333fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f333410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f3333fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f333410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44610/0 (socket says 172.21.15.83:44610) 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f3333fff700 1 -- 172.21.15.83:0/2912278009 learned_addr learned my addr 172.21.15.83:0/2912278009 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f3333fff700 1 -- 172.21.15.83:0/2912278009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332400b200 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.597+0000 7f3333fff700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f333410ab70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3324006b80 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324011e10 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3324011610 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333410b260 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401a7b0 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333410b6f0 con 0x7f33340703f0 2024-02-10T09:22:16.631 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f332400f070 con 0x7f33340703f0 2024-02-10T09:22:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f3330ff9700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332002ff70 0x7f3320032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.598+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3324042130 con 0x7f33340703f0 2024-02-10T09:22:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.599+0000 7f33337fe700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332002ff70 0x7f3320032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.599+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3334045ad0 con 0x7f33340703f0 2024-02-10T09:22:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.599+0000 7f33337fe700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332002ff70 0x7f3320032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f33280096a0 tx=0x7f3328006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:16.632 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.603+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3324016070 con 0x7f33340703f0 2024-02-10T09:22:16.770 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.769+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f3334104c40 con 0x7f332002ff70 2024-02-10T09:22:16.935 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.934+0000 7f3330ff9700 1 -- 172.21.15.83:0/2912278009 <== mgr.14162 v2:172.21.15.83:6800/1369708861 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f3334104c40 con 0x7f332002ff70 2024-02-10T09:22:16.935 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332002ff70 msgr2=0x7f3320032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332002ff70 0x7f3320032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f33280096a0 tx=0x7f3328006b40).stop 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 msgr2=0x7f333410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f333410ab70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f3324006b80 tx=0x7f3324006ee0).stop 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 shutdown_connections 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332002ff70 0x7f3320032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 --2- 172.21.15.83:0/2912278009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33340703f0 0x7f333410ab70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.936+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 >> 172.21.15.83:0/2912278009 conn(0x7f33340f6530 msgr2=0x7f33340f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.937+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 shutdown_connections 2024-02-10T09:22:16.938 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:16.937+0000 7f333a4c6700 1 -- 172.21.15.83:0/2912278009 wait complete. 2024-02-10T09:22:17.113 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:16.043319+0000 mon.smithi083 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi083.dowdym(active, since 2s) 2024-02-10T09:22:17.113 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:16.234344+0000 mon.smithi083 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:17.113 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:16.770804+0000 mgr.smithi083.dowdym (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:17.113 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:16.933194+0000 mon.smithi083 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:17.113 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:16.934995+0000 mon.smithi083 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:17.852 INFO:teuthology.orchestra.run.smithi083.stderr:Creating initial admin user... 2024-02-10T09:22:17.852 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpu4z4_d17:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-10T09:22:18.441 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.433+0000 7fc4f7167700 1 Processor -- start 2024-02-10T09:22:18.441 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f7167700 1 -- start start 2024-02-10T09:22:18.441 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f7167700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f0100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:18.441 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f7167700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00fc880 con 0x7fc4f00703f0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f4f03700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f0100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f4f03700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f0100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44620/0 (socket says 172.21.15.83:44620) 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f4f03700 1 -- 172.21.15.83:0/3705946941 learned_addr learned my addr 172.21.15.83:0/3705946941 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f4f03700 1 -- 172.21.15.83:0/3705946941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f0101380 con 0x7fc4f00703f0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4f4f03700 1 --2- 172.21.15.83:0/3705946941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f0100e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc4e400b5f0 tx=0x7fc4e4006ee0).ready entity=mon.0 client_cookie=4cb6d741bd0212b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4ef7fe700 1 -- 172.21.15.83:0/3705946941 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4e400d600 con 0x7fc4f00703f0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4ef7fe700 1 -- 172.21.15.83:0/3705946941 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4e400dc30 con 0x7fc4f00703f0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.434+0000 7fc4ef7fe700 1 -- 172.21.15.83:0/3705946941 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4e4011d60 con 0x7fc4f00703f0 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 -- 172.21.15.83:0/3705946941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 msgr2=0x7fc4f0100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 --2- 172.21.15.83:0/3705946941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f0100e90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc4e400b5f0 tx=0x7fc4e4006ee0).stop 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 -- 172.21.15.83:0/3705946941 shutdown_connections 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 --2- 172.21.15.83:0/3705946941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f0100e90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 -- 172.21.15.83:0/3705946941 >> 172.21.15.83:0/3705946941 conn(0x7fc4f00f6530 msgr2=0x7fc4f00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:18.442 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 -- 172.21.15.83:0/3705946941 shutdown_connections 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 -- 172.21.15.83:0/3705946941 wait complete. 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 Processor -- start 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.435+0000 7fc4f7167700 1 -- start start 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f7167700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f010ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f7167700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f010d3a0 con 0x7fc4f00703f0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f4f03700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f010ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f4f03700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f010ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44622/0 (socket says 172.21.15.83:44622) 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f4f03700 1 -- 172.21.15.83:0/745520894 learned_addr learned my addr 172.21.15.83:0/745520894 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f4f03700 1 -- 172.21.15.83:0/745520894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e400b200 con 0x7fc4f00703f0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4f4f03700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f010ceb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fc4e400e040 tx=0x7fc4e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.436+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4e4011e10 con 0x7fc4f00703f0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f010d5a0 con 0x7fc4f00703f0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4e401b8e0 con 0x7fc4f00703f0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4e401a930 con 0x7fc4f00703f0 2024-02-10T09:22:18.443 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f010e150 con 0x7fc4f00703f0 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc4e400f070 con 0x7fc4f00703f0 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4ed7fa700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc4d402ff70 0x7fc4d4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.437+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4e40410d0 con 0x7fc4f00703f0 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.438+0000 7fc4effff700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc4d402ff70 0x7fc4d4032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.438+0000 7fc4effff700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc4d402ff70 0x7fc4d4032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc4dc0096a0 tx=0x7fc4dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.438+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0059100 con 0x7fc4f00703f0 2024-02-10T09:22:18.444 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.442+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4e4016070 con 0x7fc4f00703f0 2024-02-10T09:22:18.611 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.609+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc4f0058f20 con 0x7fc4d402ff70 2024-02-10T09:22:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.831+0000 7fc4ed7fa700 1 -- 172.21.15.83:0/745520894 <== mgr.14162 v2:172.21.15.83:6800/1369708861 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fc4f0058f20 con 0x7fc4d402ff70 2024-02-10T09:22:18.833 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$BfK4tUCqipWSW8aKk2o9PeARhBH936Cy.CoqT4cg2qUe7bdlrcbIe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707556938, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T09:22:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc4d402ff70 msgr2=0x7fc4d4032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:18.835 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc4d402ff70 0x7fc4d4032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc4dc0096a0 tx=0x7fc4dc006b40).stop 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 msgr2=0x7fc4f010ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f010ceb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fc4e400e040 tx=0x7fc4e4006ee0).stop 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 shutdown_connections 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc4d402ff70 0x7fc4d4032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 --2- 172.21.15.83:0/745520894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4f00703f0 0x7fc4f010ceb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 >> 172.21.15.83:0/745520894 conn(0x7fc4f00f6530 msgr2=0x7fc4f00f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 shutdown_connections 2024-02-10T09:22:18.836 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:18.833+0000 7fc4f7167700 1 -- 172.21.15.83:0/745520894 wait complete. 2024-02-10T09:22:19.572 INFO:teuthology.orchestra.run.smithi083.stderr:Fetching dashboard port number... 2024-02-10T09:22:19.572 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpnrnufnem:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpcc4im29b:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.175+0000 7ff04c8a6700 1 Processor -- start 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.175+0000 7ff04c8a6700 1 -- start start 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.175+0000 7ff04c8a6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.175+0000 7ff04c8a6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0480fc7b0 con 0x7ff0480fbea0 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44628/0 (socket says 172.21.15.83:44628) 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04659c700 1 -- 172.21.15.83:0/3415862706 learned_addr learned my addr 172.21.15.83:0/3415862706 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04659c700 1 -- 172.21.15.83:0/3415862706 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480fc8f0 con 0x7ff0480fbea0 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04659c700 1 --2- 172.21.15.83:0/3415862706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480fc2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff03c00b5f0 tx=0x7ff03c006ee0).ready entity=mon.0 client_cookie=83ecd0f05161883d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:20.179 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04559a700 1 -- 172.21.15.83:0/3415862706 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03c00d600 con 0x7ff0480fbea0 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04559a700 1 -- 172.21.15.83:0/3415862706 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff03c00dc30 con 0x7ff0480fbea0 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.176+0000 7ff04559a700 1 -- 172.21.15.83:0/3415862706 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03c011d30 con 0x7ff0480fbea0 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 -- 172.21.15.83:0/3415862706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 msgr2=0x7ff0480fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 --2- 172.21.15.83:0/3415862706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480fc2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff03c00b5f0 tx=0x7ff03c006ee0).stop 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 -- 172.21.15.83:0/3415862706 shutdown_connections 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 --2- 172.21.15.83:0/3415862706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480fc2c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:20.180 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 -- 172.21.15.83:0/3415862706 >> 172.21.15.83:0/3415862706 conn(0x7ff0480f76a0 msgr2=0x7ff0480f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:20.182 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 -- 172.21.15.83:0/3415862706 shutdown_connections 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.177+0000 7ff04c8a6700 1 -- 172.21.15.83:0/3415862706 wait complete. 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04c8a6700 1 Processor -- start 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04c8a6700 1 -- start start 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04c8a6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04c8a6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048070a60 con 0x7ff0480fbea0 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44642/0 (socket says 172.21.15.83:44642) 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.178+0000 7ff04659c700 1 -- 172.21.15.83:0/2736564383 learned_addr learned my addr 172.21.15.83:0/2736564383 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff04659c700 1 -- 172.21.15.83:0/2736564383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c00b200 con 0x7ff0480fbea0 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff04659c700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480725e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff03c000c00 tx=0x7ff03c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03c00d9e0 con 0x7ff0480fbea0 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff03c01b8e0 con 0x7ff0480fbea0 2024-02-10T09:22:20.183 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff048070c00 con 0x7ff0480fbea0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff03c01a7f0 con 0x7ff0480fbea0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.179+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff048071090 con 0x7ff0480fbea0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.180+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff03c00f070 con 0x7ff0480fbea0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.180+0000 7ff036ffd700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff02c02ff70 0x7ff02c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.180+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff03c041640 con 0x7ff0480fbea0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.180+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff048045ad0 con 0x7ff0480fbea0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.180+0000 7ff045d9b700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff02c02ff70 0x7ff02c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:20.184 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.180+0000 7ff045d9b700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff02c02ff70 0x7ff02c032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff0380096a0 tx=0x7ff038006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:20.185 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.184+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff03c019cd0 con 0x7ff0480fbea0 2024-02-10T09:22:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:18.611666+0000 mgr.smithi083.dowdym (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:18.833078+0000 mon.smithi083 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:20.336 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.335+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff048104c20 con 0x7ff0480fbea0 2024-02-10T09:22:20.337 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.335+0000 7ff036ffd700 1 -- 172.21.15.83:0/2736564383 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff03c019af0 con 0x7ff0480fbea0 2024-02-10T09:22:20.337 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 8443 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff02c02ff70 msgr2=0x7ff02c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff02c02ff70 0x7ff02c032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff0380096a0 tx=0x7ff038006b40).stop 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 msgr2=0x7ff0480725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480725e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff03c000c00 tx=0x7ff03c006ee0).stop 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 shutdown_connections 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff02c02ff70 0x7ff02c032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:20.339 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 --2- 172.21.15.83:0/2736564383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0480fbea0 0x7ff0480725e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:20.340 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 >> 172.21.15.83:0/2736564383 conn(0x7ff0480f76a0 msgr2=0x7ff0480f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:20.340 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 shutdown_connections 2024-02-10T09:22:20.340 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/ceph: 2024-02-10T09:22:20.337+0000 7ff04c8a6700 1 -- 172.21.15.83:0/2736564383 wait complete. 2024-02-10T09:22:20.946 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T09:22:20.953 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: disabled 2024-02-10T09:22:20.954 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T09:22:20.961 INFO:teuthology.orchestra.run.smithi083.stderr:systemctl: inactive 2024-02-10T09:22:20.961 INFO:teuthology.orchestra.run.smithi083.stderr:firewalld.service is not enabled 2024-02-10T09:22:20.961 INFO:teuthology.orchestra.run.smithi083.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr:Ceph Dashboard is now available at: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: URL: https://smithi083.front.sepia.ceph.com:8443/ 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: User: admin 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: Password: o8cahwp7rl 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr:You can access the Ceph CLI with: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T09:22:20.965 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr: ceph telemetry on 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr:For more information see: 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr:Bootstrap complete. 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr:Releasing lock 139768093669864 on /run/cephadm/a847c16e-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:22:20.966 INFO:teuthology.orchestra.run.smithi083.stderr:Lock 139768093669864 released on /run/cephadm/a847c16e-c7f5-11ee-95b8-87774f69a715.lock 2024-02-10T09:22:20.983 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:20 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:20.336897+0000 mon.smithi083 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.83:0/2736564383' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T09:22:20.998 INFO:tasks.cephadm:Fetching config... 2024-02-10T09:22:20.998 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:22:20.999 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T09:22:21.018 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T09:22:21.018 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:22:21.018 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T09:22:21.076 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T09:22:21.077 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:22:21.077 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/mon.smithi083/keyring of=/dev/stdout 2024-02-10T09:22:21.144 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T09:22:21.144 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-02-10T09:22:21.144 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T09:22:21.202 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T09:22:21.202 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV2fnWQMWRK1PgFJGstXW857w0XOMxdml4eb4+5LDxzWZL8ID5Fd4Av/O6J8X1goewjCM0qJS9VQIb94aIlSlmjyqagMsIeLHxPrq8lCmAUwNFW6Px28uMVx3a5S8+PLDigtyK5yjhC8PNlyWdJ7GyZ94JGm3JC1S3rCobvsrZ1raKGnDt69cW7UGiciWHfDoOEyUoWiP+HfyCLeWQh8HHAh76UdjrN0QS8PHbzfryA0igmdJZdrT4KdLp6PX40JtFefABlcPGZPPQZNV96gODtbk/Dq6hEkITgh+K9jDj3eNtpAkth4ufO9Ru9L/PdkQFnrRDKFzjFXjmPnxVeVu0UFm8x6JV69K262ltlPi8z1k8a68Ab+GCSyxlEHpIqoKNJ80o+xMhn4eJD/PjdMiXGh6WTF4fV2BfOuHaW1Jit8hu7gczCFOt9lP0hoBtTGZpA73qGz4cmHUwqaGgdLUBo+pSoMlpSKWP6IoxfRL6n7z5eI8rfRv9PMZ2bBAy8R0= ceph-a847c16e-c7f5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T09:22:21.283 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV2fnWQMWRK1PgFJGstXW857w0XOMxdml4eb4+5LDxzWZL8ID5Fd4Av/O6J8X1goewjCM0qJS9VQIb94aIlSlmjyqagMsIeLHxPrq8lCmAUwNFW6Px28uMVx3a5S8+PLDigtyK5yjhC8PNlyWdJ7GyZ94JGm3JC1S3rCobvsrZ1raKGnDt69cW7UGiciWHfDoOEyUoWiP+HfyCLeWQh8HHAh76UdjrN0QS8PHbzfryA0igmdJZdrT4KdLp6PX40JtFefABlcPGZPPQZNV96gODtbk/Dq6hEkITgh+K9jDj3eNtpAkth4ufO9Ru9L/PdkQFnrRDKFzjFXjmPnxVeVu0UFm8x6JV69K262ltlPi8z1k8a68Ab+GCSyxlEHpIqoKNJ80o+xMhn4eJD/PjdMiXGh6WTF4fV2BfOuHaW1Jit8hu7gczCFOt9lP0hoBtTGZpA73qGz4cmHUwqaGgdLUBo+pSoMlpSKWP6IoxfRL6n7z5eI8rfRv9PMZ2bBAy8R0= ceph-a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:22:21.295 DEBUG:teuthology.orchestra.run.smithi158:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV2fnWQMWRK1PgFJGstXW857w0XOMxdml4eb4+5LDxzWZL8ID5Fd4Av/O6J8X1goewjCM0qJS9VQIb94aIlSlmjyqagMsIeLHxPrq8lCmAUwNFW6Px28uMVx3a5S8+PLDigtyK5yjhC8PNlyWdJ7GyZ94JGm3JC1S3rCobvsrZ1raKGnDt69cW7UGiciWHfDoOEyUoWiP+HfyCLeWQh8HHAh76UdjrN0QS8PHbzfryA0igmdJZdrT4KdLp6PX40JtFefABlcPGZPPQZNV96gODtbk/Dq6hEkITgh+K9jDj3eNtpAkth4ufO9Ru9L/PdkQFnrRDKFzjFXjmPnxVeVu0UFm8x6JV69K262ltlPi8z1k8a68Ab+GCSyxlEHpIqoKNJ80o+xMhn4eJD/PjdMiXGh6WTF4fV2BfOuHaW1Jit8hu7gczCFOt9lP0hoBtTGZpA73qGz4cmHUwqaGgdLUBo+pSoMlpSKWP6IoxfRL6n7z5eI8rfRv9PMZ2bBAy8R0= ceph-a847c16e-c7f5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T09:22:21.342 INFO:teuthology.orchestra.run.smithi158.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV2fnWQMWRK1PgFJGstXW857w0XOMxdml4eb4+5LDxzWZL8ID5Fd4Av/O6J8X1goewjCM0qJS9VQIb94aIlSlmjyqagMsIeLHxPrq8lCmAUwNFW6Px28uMVx3a5S8+PLDigtyK5yjhC8PNlyWdJ7GyZ94JGm3JC1S3rCobvsrZ1raKGnDt69cW7UGiciWHfDoOEyUoWiP+HfyCLeWQh8HHAh76UdjrN0QS8PHbzfryA0igmdJZdrT4KdLp6PX40JtFefABlcPGZPPQZNV96gODtbk/Dq6hEkITgh+K9jDj3eNtpAkth4ufO9Ru9L/PdkQFnrRDKFzjFXjmPnxVeVu0UFm8x6JV69K262ltlPi8z1k8a68Ab+GCSyxlEHpIqoKNJ80o+xMhn4eJD/PjdMiXGh6WTF4fV2BfOuHaW1Jit8hu7gczCFOt9lP0hoBtTGZpA73qGz4cmHUwqaGgdLUBo+pSoMlpSKWP6IoxfRL6n7z5eI8rfRv9PMZ2bBAy8R0= ceph-a847c16e-c7f5-11ee-95b8-87774f69a715 2024-02-10T09:22:21.354 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T09:22:22.095 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:20.837451+0000 mon.smithi083 (mon.0) 112 : cluster [DBG] mgrmap e17: smithi083.dowdym(active, since 6s) 2024-02-10T09:22:24.227 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.224+0000 7ff7974d8700 1 -- 172.21.15.83:0/3104695141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 msgr2=0x7ff7900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.224+0000 7ff7974d8700 1 --2- 172.21.15.83:0/3104695141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7900f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7ff78c009fb0 tx=0x7ff78c009b30).stop 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.224+0000 7ff7974d8700 1 -- 172.21.15.83:0/3104695141 shutdown_connections 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.224+0000 7ff7974d8700 1 --2- 172.21.15.83:0/3104695141 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7900f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.224+0000 7ff7974d8700 1 -- 172.21.15.83:0/3104695141 >> 172.21.15.83:0/3104695141 conn(0x7ff7900f1c70 msgr2=0x7ff7900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.224+0000 7ff7974d8700 1 -- 172.21.15.83:0/3104695141 shutdown_connections 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff7974d8700 1 -- 172.21.15.83:0/3104695141 wait complete. 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff7974d8700 1 Processor -- start 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff7974d8700 1 -- start start 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff7974d8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7901067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:24.228 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff7974d8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff790106ca0 con 0x7ff7900f6510 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff795274700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7901067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff795274700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7901067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:34756/0 (socket says 172.21.15.83:34756) 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.225+0000 7ff795274700 1 -- 172.21.15.83:0/93247734 learned_addr learned my addr 172.21.15.83:0/93247734 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff795274700 1 -- 172.21.15.83:0/93247734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c005040 con 0x7ff7900f6510 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff795274700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7901067b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff78c005010 tx=0x7ff78c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff78c009320 con 0x7ff7900f6510 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff790106ea0 con 0x7ff7900f6510 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff78c013380 con 0x7ff7900f6510 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff78c014e50 con 0x7ff7900f6510 2024-02-10T09:22:24.229 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.226+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff790107330 con 0x7ff7900f6510 2024-02-10T09:22:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.227+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff78c0263a0 con 0x7ff7900f6510 2024-02-10T09:22:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.227+0000 7ff785ffb700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff77c02ff10 0x7ff77c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.227+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff78c040ec0 con 0x7ff7900f6510 2024-02-10T09:22:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.227+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff790045ad0 con 0x7ff7900f6510 2024-02-10T09:22:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.227+0000 7ff794a73700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff77c02ff10 0x7ff77c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:24.230 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.228+0000 7ff794a73700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff77c02ff10 0x7ff77c0323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff7800096a0 tx=0x7ff780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:24.233 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.231+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff78c012920 con 0x7ff7900f6510 2024-02-10T09:22:24.389 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.388+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7ff7901080e0 con 0x7ff7900f6510 2024-02-10T09:22:24.392 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.391+0000 7ff785ffb700 1 -- 172.21.15.83:0/93247734 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff78c02b020 con 0x7ff7900f6510 2024-02-10T09:22:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff77c02ff10 msgr2=0x7ff77c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:24.394 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff77c02ff10 0x7ff77c0323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7ff7800096a0 tx=0x7ff780006b40).stop 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 msgr2=0x7ff7901067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7901067b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7ff78c005010 tx=0x7ff78c00b040).stop 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 shutdown_connections 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff77c02ff10 0x7ff77c0323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 --2- 172.21.15.83:0/93247734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7900f6510 0x7ff7901067b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 >> 172.21.15.83:0/93247734 conn(0x7ff7900f1c70 msgr2=0x7ff7900f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 shutdown_connections 2024-02-10T09:22:24.395 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:24.393+0000 7ff7974d8700 1 -- 172.21.15.83:0/93247734 wait complete. 2024-02-10T09:22:24.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:23.421042+0000 mon.smithi083 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:24.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:23.694315+0000 mon.smithi083 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:24.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:22:23.700706+0000 mgr.smithi083.dowdym (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi083 on smithi083 2024-02-10T09:22:24.727 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:24.392758+0000 mon.smithi083 (mon.0) 115 : audit [INF] from='client.? 172.21.15.83:0/93247734' entity='client.admin' 2024-02-10T09:22:25.306 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi158 2024-02-10T09:22:25.306 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:22:25.306 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/etc/ceph/ceph.conf 2024-02-10T09:22:25.328 DEBUG:teuthology.orchestra.run.smithi158:> set -ex 2024-02-10T09:22:25.329 DEBUG:teuthology.orchestra.run.smithi158:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T09:22:25.389 INFO:tasks.cephadm:Adding host smithi158 to orchestrator... 2024-02-10T09:22:25.390 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph orch host add smithi158 2024-02-10T09:22:36.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:34.035838+0000 mgr.smithi083.dowdym (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:38.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:36.036103+0000 mgr.smithi083.dowdym (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:40.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:38.036467+0000 mgr.smithi083.dowdym (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:42.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:40.036824+0000 mgr.smithi083.dowdym (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:44.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:42.037153+0000 mgr.smithi083.dowdym (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:46.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:44.037414+0000 mgr.smithi083.dowdym (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:46.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 -- 172.21.15.158:0/4274975015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 msgr2=0x7f4d140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:46.958 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 --2- 172.21.15.158:0/4274975015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4d04009960 tx=0x7f4d04004d10).stop 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 -- 172.21.15.158:0/4274975015 shutdown_connections 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 --2- 172.21.15.158:0/4274975015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 -- 172.21.15.158:0/4274975015 >> 172.21.15.158:0/4274975015 conn(0x7f4d140f09a0 msgr2=0x7f4d140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 -- 172.21.15.158:0/4274975015 shutdown_connections 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 -- 172.21.15.158:0/4274975015 wait complete. 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.954+0000 7f4d19ad3700 1 Processor -- start 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d19ad3700 1 -- start start 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d19ad3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d19ad3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d140f13e0 con 0x7f4d140f51a0 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d18ad1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d18ad1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56804/0 (socket says 172.21.15.158:56804) 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d18ad1700 1 -- 172.21.15.158:0/2782182649 learned_addr learned my addr 172.21.15.158:0/2782182649 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d18ad1700 1 -- 172.21.15.158:0/2782182649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d04009640 con 0x7f4d140f51a0 2024-02-10T09:22:46.959 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.955+0000 7f4d18ad1700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f2f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4d04007bd0 tx=0x7f4d04004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d04018070 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d0400b970 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d140f1580 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d0401d600 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d140f1a10 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d04024060 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.956+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d14059100 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.957+0000 7f4d117fa700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4d0002ff60 0x7f4d00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.957+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d040411f0 con 0x7f4d140f51a0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.957+0000 7f4d13fff700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4d0002ff60 0x7f4d00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:46.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.957+0000 7f4d13fff700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4d0002ff60 0x7f4d00032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4cfc0096a0 tx=0x7f4cfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:46.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:46.961+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d040116d0 con 0x7f4d140f51a0 2024-02-10T09:22:47.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.133+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi158", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d14100200 con 0x7f4d0002ff60 2024-02-10T09:22:47.731 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.729+0000 7f4d117fa700 1 -- 172.21.15.158:0/2782182649 <== mgr.14162 v2:172.21.15.83:6800/1369708861 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f4d14100200 con 0x7f4d0002ff60 2024-02-10T09:22:47.731 INFO:teuthology.orchestra.run.smithi158.stdout:Added host 'smithi158' 2024-02-10T09:22:47.732 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4d0002ff60 msgr2=0x7f4d00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:47.732 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4d0002ff60 0x7f4d00032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4cfc0096a0 tx=0x7f4cfc006b40).stop 2024-02-10T09:22:47.732 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 msgr2=0x7f4d140f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:47.732 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f2f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4d04007bd0 tx=0x7f4d04004d10).stop 2024-02-10T09:22:47.732 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 shutdown_connections 2024-02-10T09:22:47.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4d0002ff60 0x7f4d00032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:47.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 --2- 172.21.15.158:0/2782182649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d140f51a0 0x7f4d140f2f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:47.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 >> 172.21.15.158:0/2782182649 conn(0x7f4d140f09a0 msgr2=0x7f4d1405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:47.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 shutdown_connections 2024-02-10T09:22:47.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:47.731+0000 7f4d19ad3700 1 -- 172.21.15.158:0/2782182649 wait complete. 2024-02-10T09:22:48.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:46.037813+0000 mgr.smithi083.dowdym (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:48.397 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T09:22:49.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:47.135967+0000 mgr.smithi083.dowdym (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi158", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:49.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:47.729226+0000 mon.smithi083 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:49.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:22:47.729680+0000 mgr.smithi083.dowdym (mgr.14162) 14 : cephadm [INF] Added host smithi158 2024-02-10T09:22:50.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:48.038092+0000 mgr.smithi083.dowdym (mgr.14162) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 -- 172.21.15.158:0/2863630524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 msgr2=0x7f11580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 --2- 172.21.15.158:0/2863630524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f11580f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f114c009fb0 tx=0x7f114c009b30).stop 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 -- 172.21.15.158:0/2863630524 shutdown_connections 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 --2- 172.21.15.158:0/2863630524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f11580f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 -- 172.21.15.158:0/2863630524 >> 172.21.15.158:0/2863630524 conn(0x7f11580f0b70 msgr2=0x7f11580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 -- 172.21.15.158:0/2863630524 shutdown_connections 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.129+0000 7f115e708700 1 -- 172.21.15.158:0/2863630524 wait complete. 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f115e708700 1 Processor -- start 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f115e708700 1 -- start start 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f115e708700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f1158109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f115e708700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158060220 con 0x7f11580f5410 2024-02-10T09:22:50.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f1157fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f1158109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:50.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f1157fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f1158109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46666/0 (socket says 172.21.15.158:46666) 2024-02-10T09:22:50.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.130+0000 7f1157fff700 1 -- 172.21.15.158:0/1384840641 learned_addr learned my addr 172.21.15.158:0/1384840641 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f1157fff700 1 -- 172.21.15.158:0/1384840641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c005040 con 0x7f11580f5410 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f1157fff700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f1158109540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f114c005010 tx=0x7f114c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f114c009320 con 0x7f11580f5410 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1158060420 con 0x7f11580f5410 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f114c013380 con 0x7f11580f5410 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f114c014e50 con 0x7f11580f5410 2024-02-10T09:22:50.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.131+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1158109bf0 con 0x7f11580f5410 2024-02-10T09:22:50.135 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.132+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f114c0263a0 con 0x7f11580f5410 2024-02-10T09:22:50.135 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.132+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1158059100 con 0x7f11580f5410 2024-02-10T09:22:50.135 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.132+0000 7f1154ff9700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f114002ff10 0x7f11400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:50.135 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.132+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f114c040ee0 con 0x7f11580f5410 2024-02-10T09:22:50.135 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.133+0000 7f11577fe700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f114002ff10 0x7f11400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:50.135 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.133+0000 7f11577fe700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f114002ff10 0x7f11400323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f11480096a0 tx=0x7f1148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:50.138 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.138+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f114c01dd70 con 0x7f11580f5410 2024-02-10T09:22:50.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.307+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11580f1d20 con 0x7f114002ff10 2024-02-10T09:22:50.310 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.310+0000 7f1154ff9700 1 -- 172.21.15.158:0/1384840641 <== mgr.14162 v2:172.21.15.83:6800/1369708861 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f11580f1d20 con 0x7f114002ff10 2024-02-10T09:22:50.311 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:22:50.311 INFO:teuthology.orchestra.run.smithi158.stdout:[{"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}, {"addr": "smithi158", "hostname": "smithi158", "labels": [], "status": ""}] 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.310+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f114002ff10 msgr2=0x7f11400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.310+0000 7f115e708700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f114002ff10 0x7f11400323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f11480096a0 tx=0x7f1148006b40).stop 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.310+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 msgr2=0x7f1158109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.310+0000 7f115e708700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f1158109540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f114c005010 tx=0x7f114c00b040).stop 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.311+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 shutdown_connections 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.311+0000 7f115e708700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f114002ff10 0x7f11400323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:50.312 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.311+0000 7f115e708700 1 --2- 172.21.15.158:0/1384840641 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11580f5410 0x7f1158109540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:50.313 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.311+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 >> 172.21.15.158:0/1384840641 conn(0x7f11580f0b70 msgr2=0x7f11580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:50.313 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.311+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 shutdown_connections 2024-02-10T09:22:50.313 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:50.311+0000 7f115e708700 1 -- 172.21.15.158:0/1384840641 wait complete. 2024-02-10T09:22:50.916 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T09:22:50.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T09:22:52.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:50.038370+0000 mgr.smithi083.dowdym (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:52.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:50.309999+0000 mgr.smithi083.dowdym (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 -- 172.21.15.83:0/378501933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 msgr2=0x7faa000f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 --2- 172.21.15.83:0/378501933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f4be0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fa9f4009fb0 tx=0x7fa9f4009b30).stop 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 -- 172.21.15.83:0/378501933 shutdown_connections 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 --2- 172.21.15.83:0/378501933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f4be0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 -- 172.21.15.83:0/378501933 >> 172.21.15.83:0/378501933 conn(0x7faa000f02e0 msgr2=0x7faa000f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 -- 172.21.15.83:0/378501933 shutdown_connections 2024-02-10T09:22:52.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.648+0000 7faa05358700 1 -- 172.21.15.83:0/378501933 wait complete. 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7faa05358700 1 Processor -- start 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7faa05358700 1 -- start start 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7faa05358700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7faa05358700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000f57f0 con 0x7faa000f47c0 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9feffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9feffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:54082/0 (socket says 172.21.15.83:54082) 2024-02-10T09:22:52.651 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9feffd700 1 -- 172.21.15.83:0/3750946613 learned_addr learned my addr 172.21.15.83:0/3750946613 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9feffd700 1 -- 172.21.15.83:0/3750946613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4005040 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9feffd700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f7370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa9f4005010 tx=0x7fa9f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4009320 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9f4013380 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4014e50 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.649+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa000f5990 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.650+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa000f5e20 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.650+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9f40263a0 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.650+0000 7fa9e77fe700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa9e802ff10 0x7fa9e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.650+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9f4041180 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.650+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa00059100 con 0x7faa000f47c0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.650+0000 7fa9fe7fc700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa9e802ff10 0x7fa9e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:52.652 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.651+0000 7fa9fe7fc700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa9e802ff10 0x7fa9e80323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa9f00096a0 tx=0x7fa9f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:52.655 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.654+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9f4012920 con 0x7faa000f47c0 2024-02-10T09:22:52.809 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:52.808+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7faa00045ad0 con 0x7faa000f47c0 2024-02-10T09:22:53.744 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.743+0000 7fa9e77fe700 1 -- 172.21.15.83:0/3750946613 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa9f402b070 con 0x7faa000f47c0 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa9e802ff10 msgr2=0x7fa9e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa9e802ff10 0x7fa9e80323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa9f00096a0 tx=0x7fa9f0006b40).stop 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 msgr2=0x7faa000f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f7370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa9f4005010 tx=0x7fa9f400b040).stop 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 shutdown_connections 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa9e802ff10 0x7fa9e80323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:53.746 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 --2- 172.21.15.83:0/3750946613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa000f47c0 0x7faa000f7370 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:53.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.745+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 >> 172.21.15.83:0/3750946613 conn(0x7faa000f02e0 msgr2=0x7faa000f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:53.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.746+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 shutdown_connections 2024-02-10T09:22:53.747 INFO:teuthology.orchestra.run.smithi083.stderr:2024-02-10T09:22:53.746+0000 7faa05358700 1 -- 172.21.15.83:0/3750946613 wait complete. 2024-02-10T09:22:53.747 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-02-10T09:22:54.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:52.038710+0000 mgr.smithi083.dowdym (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:54.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:52.809738+0000 mon.smithi083 (mon.0) 117 : audit [INF] from='client.? 172.21.15.83:0/3750946613' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T09:22:54.422 INFO:tasks.cephadm:Adding mon.smithi083 on smithi083 2024-02-10T09:22:54.422 INFO:tasks.cephadm:Adding mon.smithi158 on smithi158 2024-02-10T09:22:54.422 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi083:172.21.15.83=smithi083;smithi158:172.21.15.158=smithi158' 2024-02-10T09:22:55.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:54 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:53.743844+0000 mon.smithi083 (mon.0) 118 : audit [INF] from='client.? 172.21.15.83:0/3750946613' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T09:22:55.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:54 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:53.743897+0000 mon.smithi083 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-10T09:22:56.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:54.038984+0000 mgr.smithi083.dowdym (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:56.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 -- 172.21.15.158:0/2633770486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 msgr2=0x7f5d7c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:56.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 --2- 172.21.15.158:0/2633770486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c0f9150 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f5d70009fb0 tx=0x7f5d70009b30).stop 2024-02-10T09:22:56.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 -- 172.21.15.158:0/2633770486 shutdown_connections 2024-02-10T09:22:56.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 --2- 172.21.15.158:0/2633770486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c0f9150 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:56.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 -- 172.21.15.158:0/2633770486 >> 172.21.15.158:0/2633770486 conn(0x7f5d7c0f0b70 msgr2=0x7f5d7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:56.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 -- 172.21.15.158:0/2633770486 shutdown_connections 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.245+0000 7f5d81b6a700 1 -- 172.21.15.158:0/2633770486 wait complete. 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d81b6a700 1 Processor -- start 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d81b6a700 1 -- start start 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d81b6a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d81b6a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c106cb0 con 0x7f5d7c060220 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d7b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d7b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46686/0 (socket says 172.21.15.158:46686) 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.246+0000 7f5d7b7fe700 1 -- 172.21.15.158:0/751921452 learned_addr learned my addr 172.21.15.158:0/751921452 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d7b7fe700 1 -- 172.21.15.158:0/751921452 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70005040 con 0x7f5d7c060220 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d7b7fe700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c1067c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5d700089f0 tx=0x7f5d7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d70009320 con 0x7f5d7c060220 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c106eb0 con 0x7f5d7c060220 2024-02-10T09:22:56.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d70013380 con 0x7f5d7c060220 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d70014e20 con 0x7f5d7c060220 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.247+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c107340 con 0x7f5d7c060220 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.248+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d70014420 con 0x7f5d7c060220 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.248+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c045ad0 con 0x7f5d7c060220 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.248+0000 7f5d63fff700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d6402ff10 0x7f5d640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.248+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d70040fc0 con 0x7f5d7c060220 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.248+0000 7f5d7affd700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d6402ff10 0x7f5d640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.249+0000 7f5d7affd700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d6402ff10 0x7f5d640323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f5d6c0096a0 tx=0x7f5d6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:56.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.253+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d7006d2b0 con 0x7f5d7c060220 2024-02-10T09:22:56.428 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.427+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 --> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi083:172.21.15.83=smithi083;smithi158:172.21.15.158=smithi158", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d7c0f1db0 con 0x7f5d6402ff10 2024-02-10T09:22:56.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.433+0000 7f5d63fff700 1 -- 172.21.15.158:0/751921452 <== mgr.14162 v2:172.21.15.83:6800/1369708861 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5d7c0f1db0 con 0x7f5d6402ff10 2024-02-10T09:22:56.433 INFO:teuthology.orchestra.run.smithi158.stdout:Scheduled mon update... 2024-02-10T09:22:56.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d6402ff10 msgr2=0x7f5d640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d6402ff10 0x7f5d640323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f5d6c0096a0 tx=0x7f5d6c006b40).stop 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 msgr2=0x7f5d7c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c1067c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5d700089f0 tx=0x7f5d7000b040).stop 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 shutdown_connections 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d6402ff10 0x7f5d640323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 --2- 172.21.15.158:0/751921452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d7c060220 0x7f5d7c1067c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 >> 172.21.15.158:0/751921452 conn(0x7f5d7c0f0b70 msgr2=0x7f5d7c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 shutdown_connections 2024-02-10T09:22:56.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:56.434+0000 7f5d81b6a700 1 -- 172.21.15.158:0/751921452 wait complete. 2024-02-10T09:22:57.113 DEBUG:teuthology.orchestra.run.smithi158:mon.smithi158> sudo journalctl -f -n 0 -u ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158.service 2024-02-10T09:22:57.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:22:57.116 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:22:57.144 INFO:journalctl@ceph.mon.smithi158.smithi158.stdout:-- Logs begin at Sat 2024-02-10 09:06:00 UTC. -- 2024-02-10T09:22:57.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:56.039261+0000 mgr.smithi083.dowdym (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:57.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:56.428852+0000 mgr.smithi083.dowdym (mgr.14162) 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi083:172.21.15.83=smithi083;smithi158:172.21.15.158=smithi158", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T09:22:57.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:22:56.430248+0000 mgr.smithi083.dowdym (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi083:172.21.15.83=smithi083;smithi158:172.21.15.158=smithi158;count:2 2024-02-10T09:22:57.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:56.433003+0000 mon.smithi083 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:22:58.825 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.823+0000 7f6c6359e700 1 -- 172.21.15.158:0/2600860577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 msgr2=0x7f6c640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:58.825 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.823+0000 7f6c6359e700 1 --2- 172.21.15.158:0/2600860577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c640f55c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6c54009fb0 tx=0x7f6c54009b30).stop 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.823+0000 7f6c6359e700 1 -- 172.21.15.158:0/2600860577 shutdown_connections 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.823+0000 7f6c6359e700 1 --2- 172.21.15.158:0/2600860577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c640f55c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.823+0000 7f6c6359e700 1 -- 172.21.15.158:0/2600860577 >> 172.21.15.158:0/2600860577 conn(0x7f6c640f09a0 msgr2=0x7f6c640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6359e700 1 -- 172.21.15.158:0/2600860577 shutdown_connections 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6359e700 1 -- 172.21.15.158:0/2600860577 wait complete. 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6359e700 1 Processor -- start 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6359e700 1 -- start start 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6359e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c641054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6359e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c641059c0 con 0x7f6c640f51a0 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c641054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c641054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46706/0 (socket says 172.21.15.158:46706) 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.824+0000 7f6c6259c700 1 -- 172.21.15.158:0/4202166981 learned_addr learned my addr 172.21.15.158:0/4202166981 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:22:58.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c6259c700 1 -- 172.21.15.158:0/4202166981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c54005040 con 0x7f6c640f51a0 2024-02-10T09:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c6259c700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c641054d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6c540089f0 tx=0x7f6c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c54009320 con 0x7f6c640f51a0 2024-02-10T09:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c64105bc0 con 0x7f6c640f51a0 2024-02-10T09:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c54013380 con 0x7f6c640f51a0 2024-02-10T09:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c54014e20 con 0x7f6c640f51a0 2024-02-10T09:22:58.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.825+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c64106050 con 0x7f6c640f51a0 2024-02-10T09:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.826+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c64045ad0 con 0x7f6c640f51a0 2024-02-10T09:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.826+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c540095f0 con 0x7f6c640f51a0 2024-02-10T09:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.826+0000 7f6c52ffd700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c4c02ff10 0x7f6c4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.826+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c54040f50 con 0x7f6c640f51a0 2024-02-10T09:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.826+0000 7f6c61d9b700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c4c02ff10 0x7f6c4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:22:58.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.827+0000 7f6c61d9b700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c4c02ff10 0x7f6c4c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f6c580096a0 tx=0x7f6c58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:22:58.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:58.830+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c54014420 con 0x7f6c640f51a0 2024-02-10T09:22:59.035 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.035+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c64106f00 con 0x7f6c640f51a0 2024-02-10T09:22:59.036 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.035+0000 7f6c52ffd700 1 -- 172.21.15.158:0/4202166981 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c54010230 con 0x7f6c640f51a0 2024-02-10T09:22:59.036 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:22:59.036 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:22:59.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c4c02ff10 msgr2=0x7f6c4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:59.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c4c02ff10 0x7f6c4c0323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f6c580096a0 tx=0x7f6c58006b40).stop 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 msgr2=0x7f6c641054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c641054d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f6c540089f0 tx=0x7f6c5400b040).stop 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 shutdown_connections 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c4c02ff10 0x7f6c4c0323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 --2- 172.21.15.158:0/4202166981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c640f51a0 0x7f6c641054d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 >> 172.21.15.158:0/4202166981 conn(0x7f6c640f09a0 msgr2=0x7f6c640f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 shutdown_connections 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:22:59.037+0000 7f6c6359e700 1 -- 172.21.15.158:0/4202166981 wait complete. 2024-02-10T09:22:59.038 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:22:59.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:22:58.039571+0000 mgr.smithi083.dowdym (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:22:59.775 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:22:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:22:59.035868+0000 mon.smithi083 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.158:0/4202166981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:00.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:00.626 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:02.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:00.039872+0000 mgr.smithi083.dowdym (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 -- 172.21.15.158:0/2164606122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 msgr2=0x7fa9840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 --2- 172.21.15.158:0/2164606122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fa978009fb0 tx=0x7fa978009b30).stop 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 -- 172.21.15.158:0/2164606122 shutdown_connections 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 --2- 172.21.15.158:0/2164606122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 -- 172.21.15.158:0/2164606122 >> 172.21.15.158:0/2164606122 conn(0x7fa9840f0b70 msgr2=0x7fa9840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 -- 172.21.15.158:0/2164606122 shutdown_connections 2024-02-10T09:23:03.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.001+0000 7fa98a79e700 1 -- 172.21.15.158:0/2164606122 wait complete. 2024-02-10T09:23:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa98a79e700 1 Processor -- start 2024-02-10T09:23:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa98a79e700 1 -- start start 2024-02-10T09:23:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa98a79e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:03.003 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa98a79e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840fb960 con 0x7fa9840f5410 2024-02-10T09:23:03.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa983fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa983fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44830/0 (socket says 172.21.15.158:44830) 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.002+0000 7fa983fff700 1 -- 172.21.15.158:0/1466942185 learned_addr learned my addr 172.21.15.158:0/1466942185 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa983fff700 1 -- 172.21.15.158:0/1466942185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978005040 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa983fff700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840fd4e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa9780089f0 tx=0x7fa97800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa978009320 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9840fbb00 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa978013380 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa97801ed80 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.003+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9840fbf90 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.004+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa97801e420 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.004+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa984059100 con 0x7fa9840f5410 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.004+0000 7fa980ff9700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa96c02ff60 0x7fa96c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:03.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.004+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9780410f0 con 0x7fa9840f5410 2024-02-10T09:23:03.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.004+0000 7fa9837fe700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa96c02ff60 0x7fa96c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:03.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.004+0000 7fa9837fe700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa96c02ff60 0x7fa96c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa9740096a0 tx=0x7fa974006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:03.009 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.009+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa97801ccf0 con 0x7fa9840f5410 2024-02-10T09:23:03.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.212+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa9840fafa0 con 0x7fa9840f5410 2024-02-10T09:23:03.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.212+0000 7fa980ff9700 1 -- 172.21.15.158:0/1466942185 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa97801cb10 con 0x7fa9840f5410 2024-02-10T09:23:03.213 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:03.213 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:03.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa96c02ff60 msgr2=0x7fa96c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:03.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa96c02ff60 0x7fa96c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fa9740096a0 tx=0x7fa974006b40).stop 2024-02-10T09:23:03.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 msgr2=0x7fa9840fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840fd4e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa9780089f0 tx=0x7fa97800b040).stop 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 shutdown_connections 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa96c02ff60 0x7fa96c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 --2- 172.21.15.158:0/1466942185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa9840f5410 0x7fa9840fd4e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 >> 172.21.15.158:0/1466942185 conn(0x7fa9840f0b70 msgr2=0x7fa98405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 shutdown_connections 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:03.214+0000 7fa98a79e700 1 -- 172.21.15.158:0/1466942185 wait complete. 2024-02-10T09:23:03.215 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:04.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:02.040156+0000 mgr.smithi083.dowdym (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:04.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:03.212933+0000 mon.smithi083 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.158:0/1466942185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:04.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:04.821 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:06.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:04.040417+0000 mgr.smithi083.dowdym (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- 172.21.15.158:0/120038013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 msgr2=0x7fbd100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 --2- 172.21.15.158:0/120038013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd100f55c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fbd04009fb0 tx=0x7fbd04009b30).stop 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- 172.21.15.158:0/120038013 shutdown_connections 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 --2- 172.21.15.158:0/120038013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd100f55c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- 172.21.15.158:0/120038013 >> 172.21.15.158:0/120038013 conn(0x7fbd100f09a0 msgr2=0x7fbd100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- 172.21.15.158:0/120038013 shutdown_connections 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- 172.21.15.158:0/120038013 wait complete. 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 Processor -- start 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- start start 2024-02-10T09:23:06.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd10106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.547+0000 7fbd0f59e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd10106b40 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd10106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd10106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44848/0 (socket says 172.21.15.158:44848) 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0e59c700 1 -- 172.21.15.158:0/3721398614 learned_addr learned my addr 172.21.15.158:0/3721398614 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0e59c700 1 -- 172.21.15.158:0/3721398614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04005040 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0e59c700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd10106650 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fbd04009f80 tx=0x7fbd0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd040095f0 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd10106d40 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd04007e90 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd04014d80 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.548+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd101071d0 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.549+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd04025070 con 0x7fbd100f51a0 2024-02-10T09:23:06.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.549+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10045ad0 con 0x7fbd100f51a0 2024-02-10T09:23:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.549+0000 7fbcfeffd700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcf802ff60 0x7fbcf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.549+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd04040830 con 0x7fbd100f51a0 2024-02-10T09:23:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.549+0000 7fbd0dd9b700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcf802ff60 0x7fbcf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.550+0000 7fbd0dd9b700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcf802ff60 0x7fbcf8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fbd000096a0 tx=0x7fbd00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:06.558 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.554+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd0400f4e0 con 0x7fbd100f51a0 2024-02-10T09:23:06.760 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.760+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd10108080 con 0x7fbd100f51a0 2024-02-10T09:23:06.761 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.761+0000 7fbcfeffd700 1 -- 172.21.15.158:0/3721398614 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd0400f300 con 0x7fbd100f51a0 2024-02-10T09:23:06.761 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:06.761 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcf802ff60 msgr2=0x7fbcf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcf802ff60 0x7fbcf8032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fbd000096a0 tx=0x7fbd00006b40).stop 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 msgr2=0x7fbd10106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd10106650 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fbd04009f80 tx=0x7fbd0400b040).stop 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 shutdown_connections 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcf802ff60 0x7fbcf8032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:06.763 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 --2- 172.21.15.158:0/3721398614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd100f51a0 0x7fbd10106650 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 >> 172.21.15.158:0/3721398614 conn(0x7fbd100f09a0 msgr2=0x7fbd1005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 shutdown_connections 2024-02-10T09:23:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:06.763+0000 7fbd0f59e700 1 -- 172.21.15.158:0/3721398614 wait complete. 2024-02-10T09:23:06.764 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:08.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:06.040780+0000 mgr.smithi083.dowdym (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:08.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:06.761288+0000 mon.smithi083 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.158:0/3721398614' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:08.370 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:08.370 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:10.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:08.041149+0000 mgr.smithi083.dowdym (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:11.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 -- 172.21.15.158:0/1903705732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 msgr2=0x7f90140f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:11.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 --2- 172.21.15.158:0/1903705732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f90140f66c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9004009fb0 tx=0x7f9004009b30).stop 2024-02-10T09:23:11.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 -- 172.21.15.158:0/1903705732 shutdown_connections 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 --2- 172.21.15.158:0/1903705732 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f90140f66c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 -- 172.21.15.158:0/1903705732 >> 172.21.15.158:0/1903705732 conn(0x7f90140f1ac0 msgr2=0x7f90140f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 -- 172.21.15.158:0/1903705732 shutdown_connections 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 -- 172.21.15.158:0/1903705732 wait complete. 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 Processor -- start 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.683+0000 7f90197a8700 1 -- start start 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f90197a8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f901406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f90197a8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9014069e20 con 0x7f90140f62a0 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9013fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f901406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9013fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f901406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:38908/0 (socket says 172.21.15.158:38908) 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9013fff700 1 -- 172.21.15.158:0/937029487 learned_addr learned my addr 172.21.15.158:0/937029487 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9013fff700 1 -- 172.21.15.158:0/937029487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004005040 con 0x7f90140f62a0 2024-02-10T09:23:11.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9013fff700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f901406b9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9004009830 tx=0x7f900400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9004009320 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9014069fc0 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9004013380 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.684+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9004014420 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.685+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901406a450 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.685+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f900401e420 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.685+0000 7f9010ff9700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f900002ff60 0x7f9000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.685+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9004040ee0 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.685+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9014059100 con 0x7f90140f62a0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.686+0000 7f90137fe700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f900002ff60 0x7f9000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:11.687 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.686+0000 7f90137fe700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f900002ff60 0x7f9000032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8ffc0096a0 tx=0x7f8ffc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:11.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.690+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f900401c970 con 0x7f90140f62a0 2024-02-10T09:23:11.894 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.894+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9014045ad0 con 0x7f90140f62a0 2024-02-10T09:23:11.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.895+0000 7f9010ff9700 1 -- 172.21.15.158:0/937029487 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9014045ad0 con 0x7f90140f62a0 2024-02-10T09:23:11.896 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:11.896 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:11.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f900002ff60 msgr2=0x7f9000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:11.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f900002ff60 0x7f9000032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8ffc0096a0 tx=0x7f8ffc006b40).stop 2024-02-10T09:23:11.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 msgr2=0x7f901406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:11.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f901406b9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9004009830 tx=0x7f900400b040).stop 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 shutdown_connections 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f900002ff60 0x7f9000032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 --2- 172.21.15.158:0/937029487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90140f62a0 0x7f901406b9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 >> 172.21.15.158:0/937029487 conn(0x7f90140f1ac0 msgr2=0x7f90140f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 shutdown_connections 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:11.897+0000 7f90197a8700 1 -- 172.21.15.158:0/937029487 wait complete. 2024-02-10T09:23:11.898 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:12.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:10.041485+0000 mgr.smithi083.dowdym (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:13.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:12 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:11.895513+0000 mon.smithi083 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.158:0/937029487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:13.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:13.362 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:14.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:12.041807+0000 mgr.smithi083.dowdym (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:15.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:14.165844+0000 mon.smithi083 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:23:15.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:14.174288+0000 mon.smithi083 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:23:15.093 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 -- 172.21.15.158:0/1996999590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 msgr2=0x7f5e240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 --2- 172.21.15.158:0/1996999590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e240f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f5e14009fb0 tx=0x7f5e14009b30).stop 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 -- 172.21.15.158:0/1996999590 shutdown_connections 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 --2- 172.21.15.158:0/1996999590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e240f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 -- 172.21.15.158:0/1996999590 >> 172.21.15.158:0/1996999590 conn(0x7f5e240f0b70 msgr2=0x7f5e240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 -- 172.21.15.158:0/1996999590 shutdown_connections 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.091+0000 7f5e2b796700 1 -- 172.21.15.158:0/1996999590 wait complete. 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e2b796700 1 Processor -- start 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e2b796700 1 -- start start 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e2b796700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e241067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e2b796700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e24106cc0 con 0x7f5e240f5410 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e29532700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e241067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e29532700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e241067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:38916/0 (socket says 172.21.15.158:38916) 2024-02-10T09:23:15.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e29532700 1 -- 172.21.15.158:0/245003426 learned_addr learned my addr 172.21.15.158:0/245003426 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.092+0000 7f5e29532700 1 -- 172.21.15.158:0/245003426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e14005040 con 0x7f5e240f5410 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.093+0000 7f5e29532700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e241067d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5e14005010 tx=0x7f5e1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.093+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e14009320 con 0x7f5e240f5410 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.093+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e14013380 con 0x7f5e240f5410 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.093+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e24106ec0 con 0x7f5e240f5410 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.093+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e14014e50 con 0x7f5e240f5410 2024-02-10T09:23:15.095 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.093+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e24107350 con 0x7f5e240f5410 2024-02-10T09:23:15.096 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.094+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e14014420 con 0x7f5e240f5410 2024-02-10T09:23:15.096 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.094+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24045ad0 con 0x7f5e240f5410 2024-02-10T09:23:15.096 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.094+0000 7f5e19ffb700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e1002ff10 0x7f5e100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:15.096 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.094+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e14040e60 con 0x7f5e240f5410 2024-02-10T09:23:15.096 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.094+0000 7f5e28d31700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e1002ff10 0x7f5e100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:15.096 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.095+0000 7f5e28d31700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e1002ff10 0x7f5e100323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5e200096a0 tx=0x7f5e20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:15.099 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.099+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e140095f0 con 0x7f5e240f5410 2024-02-10T09:23:15.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.305+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e24108200 con 0x7f5e240f5410 2024-02-10T09:23:15.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.306+0000 7f5e19ffb700 1 -- 172.21.15.158:0/245003426 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e1401db70 con 0x7f5e240f5410 2024-02-10T09:23:15.306 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:15.306 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.307+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e1002ff10 msgr2=0x7f5e100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.307+0000 7f5e2b796700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e1002ff10 0x7f5e100323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5e200096a0 tx=0x7f5e20006b40).stop 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.307+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 msgr2=0x7f5e241067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.307+0000 7f5e2b796700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e241067d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f5e14005010 tx=0x7f5e1400b040).stop 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.307+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 shutdown_connections 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.307+0000 7f5e2b796700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e1002ff10 0x7f5e100323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.308+0000 7f5e2b796700 1 --2- 172.21.15.158:0/245003426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e240f5410 0x7f5e241067d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.308+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 >> 172.21.15.158:0/245003426 conn(0x7f5e240f0b70 msgr2=0x7f5e2405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.308+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 shutdown_connections 2024-02-10T09:23:15.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:15.308+0000 7f5e2b796700 1 -- 172.21.15.158:0/245003426 wait complete. 2024-02-10T09:23:15.309 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:16.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:14.042162+0000 mgr.smithi083.dowdym (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:16.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:15.306305+0000 mon.smithi083 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.158:0/245003426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:18.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:16.042532+0000 mgr.smithi083.dowdym (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:20.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:18.042842+0000 mgr.smithi083.dowdym (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:20.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:20.932 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:22.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:20.043170+0000 mgr.smithi083.dowdym (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:24.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:22.043557+0000 mgr.smithi083.dowdym (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 -- 172.21.15.158:0/2971765859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 msgr2=0x7f3fa40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 --2- 172.21.15.158:0/2971765859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa40f90f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3f8c009fb0 tx=0x7f3f8c009b30).stop 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 -- 172.21.15.158:0/2971765859 shutdown_connections 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 --2- 172.21.15.158:0/2971765859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa40f90f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 -- 172.21.15.158:0/2971765859 >> 172.21.15.158:0/2971765859 conn(0x7f3fa40f0b70 msgr2=0x7f3fa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 -- 172.21.15.158:0/2971765859 shutdown_connections 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.326+0000 7f3faa6e1700 1 -- 172.21.15.158:0/2971765859 wait complete. 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.327+0000 7f3faa6e1700 1 Processor -- start 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.327+0000 7f3faa6e1700 1 -- start start 2024-02-10T09:23:25.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.327+0000 7f3faa6e1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.327+0000 7f3faa6e1700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4106c70 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa3fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa3fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52852/0 (socket says 172.21.15.158:52852) 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa3fff700 1 -- 172.21.15.158:0/618925251 learned_addr learned my addr 172.21.15.158:0/618925251 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa3fff700 1 -- 172.21.15.158:0/618925251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c005040 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa3fff700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa4106780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f3f8c0089f0 tx=0x7f3f8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f8c009320 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa4106e70 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f8c013380 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f8c014e20 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.328+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa4107300 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.329+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f8c0263a0 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.329+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa4045ad0 con 0x7f3fa4060280 2024-02-10T09:23:25.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.329+0000 7f3fa0ff9700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3f9002ff60 0x7f3f90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:25.338 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.329+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f8c040f70 con 0x7f3fa4060280 2024-02-10T09:23:25.338 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.329+0000 7f3fa37fe700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3f9002ff60 0x7f3f90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:25.338 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.330+0000 7f3fa37fe700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3f9002ff60 0x7f3f90032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f3f940096a0 tx=0x7f3f94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:25.338 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.334+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f8c01dde0 con 0x7f3fa4060280 2024-02-10T09:23:25.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.536+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fa4108150 con 0x7f3fa4060280 2024-02-10T09:23:25.538 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.537+0000 7f3fa0ff9700 1 -- 172.21.15.158:0/618925251 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f8c010230 con 0x7f3fa4060280 2024-02-10T09:23:25.538 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:25.538 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:25.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3f9002ff60 msgr2=0x7f3f90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:25.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3f9002ff60 0x7f3f90032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f3f940096a0 tx=0x7f3f94006b40).stop 2024-02-10T09:23:25.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 msgr2=0x7f3fa4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:25.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa4106780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f3f8c0089f0 tx=0x7f3f8c00b040).stop 2024-02-10T09:23:25.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 shutdown_connections 2024-02-10T09:23:25.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3f9002ff60 0x7f3f90032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:25.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 --2- 172.21.15.158:0/618925251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fa4060280 0x7f3fa4106780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:25.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.538+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 >> 172.21.15.158:0/618925251 conn(0x7f3fa40f0b70 msgr2=0x7f3fa40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:25.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.539+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 shutdown_connections 2024-02-10T09:23:25.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:25.539+0000 7f3faa6e1700 1 -- 172.21.15.158:0/618925251 wait complete. 2024-02-10T09:23:25.541 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:26.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:24.043890+0000 mgr.smithi083.dowdym (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:26.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:25.538413+0000 mon.smithi083 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.158:0/618925251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:27.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:27.170 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:28.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:26.044145+0000 mgr.smithi083.dowdym (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:29.086 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 -- 172.21.15.158:0/3769975755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 msgr2=0x7f802c0f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:29.086 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 --2- 172.21.15.158:0/3769975755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c0f6e20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8020009fb0 tx=0x7f8020009b30).stop 2024-02-10T09:23:29.086 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 -- 172.21.15.158:0/3769975755 shutdown_connections 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 --2- 172.21.15.158:0/3769975755 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c0f6e20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 -- 172.21.15.158:0/3769975755 >> 172.21.15.158:0/3769975755 conn(0x7f802c0f0260 msgr2=0x7f802c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 -- 172.21.15.158:0/3769975755 shutdown_connections 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.084+0000 7f8032649700 1 -- 172.21.15.158:0/3769975755 wait complete. 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.085+0000 7f8032649700 1 Processor -- start 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.085+0000 7f8032649700 1 -- start start 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.085+0000 7f8032649700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:29.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.085+0000 7f8032649700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c106c60 con 0x7f802c0f6a00 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.085+0000 7f802bfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f802bfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52858/0 (socket says 172.21.15.158:52858) 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f802bfff700 1 -- 172.21.15.158:0/3774036484 learned_addr learned my addr 172.21.15.158:0/3774036484 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f802bfff700 1 -- 172.21.15.158:0/3774036484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8020005040 con 0x7f802c0f6a00 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f802bfff700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c106770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f80200089f0 tx=0x7f802000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8020009320 con 0x7f802c0f6a00 2024-02-10T09:23:29.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8020013380 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c106e60 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8020014e20 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.086+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c1072f0 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.087+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8020014420 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.087+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c045ad0 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.087+0000 7f8028ff9700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f801402ff60 0x7f8014032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.087+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8020040f20 con 0x7f802c0f6a00 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.087+0000 7f802b7fe700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f801402ff60 0x7f8014032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:29.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.088+0000 7f802b7fe700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f801402ff60 0x7f8014032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f801c0096a0 tx=0x7f801c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:29.093 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.092+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8020014790 con 0x7f802c0f6a00 2024-02-10T09:23:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.297+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f802c1081a0 con 0x7f802c0f6a00 2024-02-10T09:23:29.299 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.298+0000 7f8028ff9700 1 -- 172.21.15.158:0/3774036484 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8020010230 con 0x7f802c0f6a00 2024-02-10T09:23:29.300 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:29.300 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f801402ff60 msgr2=0x7f8014032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f801402ff60 0x7f8014032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f801c0096a0 tx=0x7f801c006b40).stop 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 msgr2=0x7f802c106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c106770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f80200089f0 tx=0x7f802000b040).stop 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 shutdown_connections 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f801402ff60 0x7f8014032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 --2- 172.21.15.158:0/3774036484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f802c0f6a00 0x7f802c106770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 >> 172.21.15.158:0/3774036484 conn(0x7f802c0f0260 msgr2=0x7f802c0f4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 shutdown_connections 2024-02-10T09:23:29.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:29.300+0000 7f8032649700 1 -- 172.21.15.158:0/3774036484 wait complete. 2024-02-10T09:23:29.303 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:30.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:28.044374+0000 mgr.smithi083.dowdym (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:30.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:29.299534+0000 mon.smithi083 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.158:0/3774036484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:30.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:30.884 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:32.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:30.044695+0000 mgr.smithi083.dowdym (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 -- 172.21.15.158:0/4155146038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 msgr2=0x7f83b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:32.744 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 --2- 172.21.15.158:0/4155146038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b40f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f83a8009fb0 tx=0x7f83a8009b30).stop 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 -- 172.21.15.158:0/4155146038 shutdown_connections 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 --2- 172.21.15.158:0/4155146038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b40f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 -- 172.21.15.158:0/4155146038 >> 172.21.15.158:0/4155146038 conn(0x7f83b40f1c70 msgr2=0x7f83b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 -- 172.21.15.158:0/4155146038 shutdown_connections 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.742+0000 7f83bb16b700 1 -- 172.21.15.158:0/4155146038 wait complete. 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83bb16b700 1 Processor -- start 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83bb16b700 1 -- start start 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83bb16b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83bb16b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b4106cd0 con 0x7f83b40f6510 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83b8f07700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83b8f07700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44854/0 (socket says 172.21.15.158:44854) 2024-02-10T09:23:32.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.743+0000 7f83b8f07700 1 -- 172.21.15.158:0/1167350235 learned_addr learned my addr 172.21.15.158:0/1167350235 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83b8f07700 1 -- 172.21.15.158:0/1167350235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a8005040 con 0x7f83b40f6510 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83b8f07700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b41067e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f83a8005010 tx=0x7f83a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a8009320 con 0x7f83b40f6510 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b4106ed0 con 0x7f83b40f6510 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b4107360 con 0x7f83b40f6510 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83a8013380 con 0x7f83b40f6510 2024-02-10T09:23:32.747 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.744+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a8014df0 con 0x7f83b40f6510 2024-02-10T09:23:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.745+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83a80095f0 con 0x7f83b40f6510 2024-02-10T09:23:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.745+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4045ad0 con 0x7f83b40f6510 2024-02-10T09:23:32.748 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.745+0000 7f83b17fa700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83a402ff10 0x7f83a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:32.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.745+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83a8040f20 con 0x7f83b40f6510 2024-02-10T09:23:32.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.745+0000 7f83b3fff700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83a402ff10 0x7f83a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:32.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.746+0000 7f83b3fff700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83a402ff10 0x7f83a40323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83a00096a0 tx=0x7f83a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:32.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.750+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83a8014420 con 0x7f83b40f6510 2024-02-10T09:23:32.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.954+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83b4108210 con 0x7f83b40f6510 2024-02-10T09:23:32.956 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.955+0000 7f83b17fa700 1 -- 172.21.15.158:0/1167350235 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83a8010230 con 0x7f83b40f6510 2024-02-10T09:23:32.959 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:32.959 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.956+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83a402ff10 msgr2=0x7f83a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.956+0000 7f83bb16b700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83a402ff10 0x7f83a40323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83a00096a0 tx=0x7f83a0006b40).stop 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.956+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 msgr2=0x7f83b41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.956+0000 7f83bb16b700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b41067e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f83a8005010 tx=0x7f83a800b040).stop 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.957+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 shutdown_connections 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.957+0000 7f83bb16b700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83a402ff10 0x7f83a40323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.957+0000 7f83bb16b700 1 --2- 172.21.15.158:0/1167350235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83b40f6510 0x7f83b41067e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:32.960 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.957+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 >> 172.21.15.158:0/1167350235 conn(0x7f83b40f1c70 msgr2=0x7f83b40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.957+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 shutdown_connections 2024-02-10T09:23:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:32.957+0000 7f83bb16b700 1 -- 172.21.15.158:0/1167350235 wait complete. 2024-02-10T09:23:32.961 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:34.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:32.044964+0000 mgr.smithi083.dowdym (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:34.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:32.956505+0000 mon.smithi083 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.158:0/1167350235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:34.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:34.565 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:36.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:34.045290+0000 mgr.smithi083.dowdym (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:36.417 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 -- 172.21.15.158:0/2712051524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 msgr2=0x7f49fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:36.417 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 --2- 172.21.15.158:0/2712051524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc0f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f49ec009fb0 tx=0x7f49ec009b30).stop 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 -- 172.21.15.158:0/2712051524 shutdown_connections 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 --2- 172.21.15.158:0/2712051524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc0f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 -- 172.21.15.158:0/2712051524 >> 172.21.15.158:0/2712051524 conn(0x7f49fc0f0b70 msgr2=0x7f49fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 -- 172.21.15.158:0/2712051524 shutdown_connections 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.403+0000 7f4a034ad700 1 -- 172.21.15.158:0/2712051524 wait complete. 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a034ad700 1 Processor -- start 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a034ad700 1 -- start start 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a034ad700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a034ad700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49fc106cd0 con 0x7f49fc0f5410 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a01249700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a01249700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44874/0 (socket says 172.21.15.158:44874) 2024-02-10T09:23:36.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.404+0000 7f4a01249700 1 -- 172.21.15.158:0/49556146 learned_addr learned my addr 172.21.15.158:0/49556146 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f4a01249700 1 -- 172.21.15.158:0/49556146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ec005040 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f4a01249700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc1067e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f49ec005010 tx=0x7f49ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49ec009320 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49fc106ed0 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49ec013380 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49ec014e50 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.405+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49fc107360 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.406+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49ec014420 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.406+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49fc045ad0 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.406+0000 7f49f1ffb700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f49e802ff10 0x7f49e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.406+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49ec040e60 con 0x7f49fc0f5410 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.406+0000 7f4a00a48700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f49e802ff10 0x7f49e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.407+0000 7f4a00a48700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f49e802ff10 0x7f49e80323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f49f80096a0 tx=0x7f49f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:36.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.411+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49ec0095f0 con 0x7f49fc0f5410 2024-02-10T09:23:36.616 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.615+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49fc108210 con 0x7f49fc0f5410 2024-02-10T09:23:36.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.616+0000 7f49f1ffb700 1 -- 172.21.15.158:0/49556146 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49ec01db70 con 0x7f49fc0f5410 2024-02-10T09:23:36.617 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:36.617 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f49e802ff10 msgr2=0x7f49e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f49e802ff10 0x7f49e80323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f49f80096a0 tx=0x7f49f8006b40).stop 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 msgr2=0x7f49fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc1067e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f49ec005010 tx=0x7f49ec00b040).stop 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 shutdown_connections 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f49e802ff10 0x7f49e80323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 --2- 172.21.15.158:0/49556146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49fc0f5410 0x7f49fc1067e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.617+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 >> 172.21.15.158:0/49556146 conn(0x7f49fc0f0b70 msgr2=0x7f49fc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.618+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 shutdown_connections 2024-02-10T09:23:36.619 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:36.618+0000 7f4a034ad700 1 -- 172.21.15.158:0/49556146 wait complete. 2024-02-10T09:23:36.620 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:37.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:36.616976+0000 mon.smithi083 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.158:0/49556146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:38.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:36.045672+0000 mgr.smithi083.dowdym (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:38.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:38.232 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:39.909 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 -- 172.21.15.158:0/699785606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 msgr2=0x7f9d600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:39.909 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 --2- 172.21.15.158:0/699785606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d600f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9d50009fb0 tx=0x7f9d50009b30).stop 2024-02-10T09:23:39.909 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 -- 172.21.15.158:0/699785606 shutdown_connections 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 --2- 172.21.15.158:0/699785606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d600f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 -- 172.21.15.158:0/699785606 >> 172.21.15.158:0/699785606 conn(0x7f9d600f1c70 msgr2=0x7f9d600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 -- 172.21.15.158:0/699785606 shutdown_connections 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 -- 172.21.15.158:0/699785606 wait complete. 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.900+0000 7f9d662f3700 1 Processor -- start 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d662f3700 1 -- start start 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d662f3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d6006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d662f3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60069e20 con 0x7f9d600f6510 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d5ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d6006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d5ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d6006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36800/0 (socket says 172.21.15.158:36800) 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d5ffff700 1 -- 172.21.15.158:0/1097471459 learned_addr learned my addr 172.21.15.158:0/1097471459 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d5ffff700 1 -- 172.21.15.158:0/1097471459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d50005040 con 0x7f9d600f6510 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.901+0000 7f9d5ffff700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d6006b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9d50005010 tx=0x7f9d5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:39.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.902+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d50009320 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.902+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d60069fc0 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.902+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d50013380 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.902+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d50014e50 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.902+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6006a450 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.903+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d60059100 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.903+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d500095f0 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.903+0000 7f9d5cff9700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9d4802ff10 0x7f9d480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.903+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d50040e90 con 0x7f9d600f6510 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.903+0000 7f9d5f7fe700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9d4802ff10 0x7f9d480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.903+0000 7f9d5f7fe700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9d4802ff10 0x7f9d480323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:39.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:39.907+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d50014420 con 0x7f9d600f6510 2024-02-10T09:23:40.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:38.045935+0000 mgr.smithi083.dowdym (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:40.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.112+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d600fc0a0 con 0x7f9d600f6510 2024-02-10T09:23:40.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.113+0000 7f9d5cff9700 1 -- 172.21.15.158:0/1097471459 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d50010230 con 0x7f9d600f6510 2024-02-10T09:23:40.114 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:40.115 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9d4802ff10 msgr2=0x7f9d480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9d4802ff10 0x7f9d480323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).stop 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 msgr2=0x7f9d6006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d6006b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9d50005010 tx=0x7f9d5000b040).stop 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 shutdown_connections 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9d4802ff10 0x7f9d480323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 --2- 172.21.15.158:0/1097471459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d600f6510 0x7f9d6006b9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.114+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 >> 172.21.15.158:0/1097471459 conn(0x7f9d600f1c70 msgr2=0x7f9d600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.115+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 shutdown_connections 2024-02-10T09:23:40.116 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:40.115+0000 7f9d662f3700 1 -- 172.21.15.158:0/1097471459 wait complete. 2024-02-10T09:23:40.117 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:41.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:40 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:40.114417+0000 mon.smithi083 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.158:0/1097471459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:41.727 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:41.728 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:42.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:40.046280+0000 mgr.smithi083.dowdym (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- 172.21.15.158:0/1323251173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 msgr2=0x7fe3d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 --2- 172.21.15.158:0/1323251173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d80f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe3cc009fb0 tx=0x7fe3cc009b30).stop 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- 172.21.15.158:0/1323251173 shutdown_connections 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 --2- 172.21.15.158:0/1323251173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d80f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- 172.21.15.158:0/1323251173 >> 172.21.15.158:0/1323251173 conn(0x7fe3d80f0b70 msgr2=0x7fe3d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- 172.21.15.158:0/1323251173 shutdown_connections 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- 172.21.15.158:0/1323251173 wait complete. 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 Processor -- start 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- start start 2024-02-10T09:23:43.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d8106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.551+0000 7fe3dcce9700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d8106d10 con 0x7fe3d80f5410 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3d659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d8106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3d659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d8106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36812/0 (socket says 172.21.15.158:36812) 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3d659c700 1 -- 172.21.15.158:0/877053828 learned_addr learned my addr 172.21.15.158:0/877053828 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3d659c700 1 -- 172.21.15.158:0/877053828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc005040 con 0x7fe3d80f5410 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3d659c700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d8106820 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe3cc005010 tx=0x7fe3cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc009320 con 0x7fe3d80f5410 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d8106f10 con 0x7fe3d80f5410 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3cc013380 con 0x7fe3d80f5410 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3cc014e50 con 0x7fe3d80f5410 2024-02-10T09:23:43.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.552+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d81073a0 con 0x7fe3d80f5410 2024-02-10T09:23:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.553+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3cc0095f0 con 0x7fe3d80f5410 2024-02-10T09:23:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.553+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d8045ad0 con 0x7fe3d80f5410 2024-02-10T09:23:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.553+0000 7fe3c6ffd700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe3c002ff10 0x7fe3c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:43.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.553+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3cc040f10 con 0x7fe3d80f5410 2024-02-10T09:23:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.553+0000 7fe3d5d9b700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe3c002ff10 0x7fe3c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:43.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.554+0000 7fe3d5d9b700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe3c002ff10 0x7fe3c00323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe3c80096a0 tx=0x7fe3c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:43.559 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.558+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3cc014420 con 0x7fe3d80f5410 2024-02-10T09:23:43.766 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.765+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3d8108250 con 0x7fe3d80f5410 2024-02-10T09:23:43.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.766+0000 7fe3c6ffd700 1 -- 172.21.15.158:0/877053828 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3cc010230 con 0x7fe3d80f5410 2024-02-10T09:23:43.767 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:43.767 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe3c002ff10 msgr2=0x7fe3c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe3c002ff10 0x7fe3c00323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe3c80096a0 tx=0x7fe3c8006b40).stop 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 msgr2=0x7fe3d8106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d8106820 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe3cc005010 tx=0x7fe3cc00b040).stop 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 shutdown_connections 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe3c002ff10 0x7fe3c00323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 --2- 172.21.15.158:0/877053828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3d80f5410 0x7fe3d8106820 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 >> 172.21.15.158:0/877053828 conn(0x7fe3d80f0b70 msgr2=0x7fe3d80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 shutdown_connections 2024-02-10T09:23:43.769 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:43.767+0000 7fe3dcce9700 1 -- 172.21.15.158:0/877053828 wait complete. 2024-02-10T09:23:43.770 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:44.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:42.046688+0000 mgr.smithi083.dowdym (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:45.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:43.767222+0000 mon.smithi083 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.158:0/877053828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:45.412 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:45.412 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:46.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:44.047037+0000 mgr.smithi083.dowdym (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/558502294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 msgr2=0x7f9ae00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 --2- 172.21.15.158:0/558502294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae00f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f9ad4009fb0 tx=0x7f9ad4009b30).stop 2024-02-10T09:23:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/558502294 shutdown_connections 2024-02-10T09:23:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 --2- 172.21.15.158:0/558502294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae00f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/558502294 >> 172.21.15.158:0/558502294 conn(0x7f9ae00f1c70 msgr2=0x7f9ae00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:47.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/558502294 shutdown_connections 2024-02-10T09:23:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.331+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/558502294 wait complete. 2024-02-10T09:23:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae6c8f700 1 Processor -- start 2024-02-10T09:23:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae6c8f700 1 -- start start 2024-02-10T09:23:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae6c8f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:47.334 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae6c8f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae0106cd0 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae4a2b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae4a2b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36834/0 (socket says 172.21.15.158:36834) 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.332+0000 7f9ae4a2b700 1 -- 172.21.15.158:0/2156261001 learned_addr learned my addr 172.21.15.158:0/2156261001 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9ae4a2b700 1 -- 172.21.15.158:0/2156261001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4005040 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9ae4a2b700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae01067e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9ad40089f0 tx=0x7f9ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad4009320 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae0106ed0 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ad4013380 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ad401ed80 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.333+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae0107360 con 0x7f9ae00f6510 2024-02-10T09:23:47.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.334+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ad401e420 con 0x7f9ae00f6510 2024-02-10T09:23:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.334+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae0045ad0 con 0x7f9ae00f6510 2024-02-10T09:23:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.334+0000 7f9add7fa700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9ad002ff60 0x7f9ad0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.334+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ad4041050 con 0x7f9ae00f6510 2024-02-10T09:23:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.334+0000 7f9adffff700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9ad002ff60 0x7f9ad0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:47.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.335+0000 7f9adffff700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9ad002ff60 0x7f9ad0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f9acc0096a0 tx=0x7f9acc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:47.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.339+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ad401cb10 con 0x7f9ae00f6510 2024-02-10T09:23:47.544 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.543+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ae0108210 con 0x7f9ae00f6510 2024-02-10T09:23:47.544 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.543+0000 7f9add7fa700 1 -- 172.21.15.158:0/2156261001 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ad4004490 con 0x7f9ae00f6510 2024-02-10T09:23:47.545 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:47.545 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:47.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9ad002ff60 msgr2=0x7f9ad0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9ad002ff60 0x7f9ad0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f9acc0096a0 tx=0x7f9acc006b40).stop 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 msgr2=0x7f9ae01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae01067e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9ad40089f0 tx=0x7f9ad400b040).stop 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 shutdown_connections 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9ad002ff60 0x7f9ad0032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 --2- 172.21.15.158:0/2156261001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae00f6510 0x7f9ae01067e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 >> 172.21.15.158:0/2156261001 conn(0x7f9ae00f1c70 msgr2=0x7f9ae00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 shutdown_connections 2024-02-10T09:23:47.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:47.545+0000 7f9ae6c8f700 1 -- 172.21.15.158:0/2156261001 wait complete. 2024-02-10T09:23:47.548 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:48.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:46.047415+0000 mgr.smithi083.dowdym (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:48.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:47.544962+0000 mon.smithi083 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.158:0/2156261001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:49.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:49.243 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:50.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:48.047833+0000 mgr.smithi083.dowdym (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 -- 172.21.15.158:0/2618376738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 msgr2=0x7fb3f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 --2- 172.21.15.158:0/2618376738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f80f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb3ec009fb0 tx=0x7fb3ec009b30).stop 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 -- 172.21.15.158:0/2618376738 shutdown_connections 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 --2- 172.21.15.158:0/2618376738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f80f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 -- 172.21.15.158:0/2618376738 >> 172.21.15.158:0/2618376738 conn(0x7fb3f80f0b70 msgr2=0x7fb3f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 -- 172.21.15.158:0/2618376738 shutdown_connections 2024-02-10T09:23:51.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.998+0000 7fb3ff01d700 1 -- 172.21.15.158:0/2618376738 wait complete. 2024-02-10T09:23:51.001 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.999+0000 7fb3ff01d700 1 Processor -- start 2024-02-10T09:23:51.001 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.999+0000 7fb3ff01d700 1 -- start start 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.999+0000 7fb3ff01d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f81094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:50.999+0000 7fb3ff01d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8060240 con 0x7fb3f80f5410 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3fcdb9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f81094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3fcdb9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f81094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40518/0 (socket says 172.21.15.158:40518) 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3fcdb9700 1 -- 172.21.15.158:0/3125088750 learned_addr learned my addr 172.21.15.158:0/3125088750 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3fcdb9700 1 -- 172.21.15.158:0/3125088750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec005040 con 0x7fb3f80f5410 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3fcdb9700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f81094e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb3ec0089f0 tx=0x7fb3ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ec009320 con 0x7fb3f80f5410 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f8060440 con 0x7fb3f80f5410 2024-02-10T09:23:51.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3ec013380 con 0x7fb3f80f5410 2024-02-10T09:23:51.003 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ec01ed80 con 0x7fb3f80f5410 2024-02-10T09:23:51.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.000+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f8109b90 con 0x7fb3f80f5410 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.001+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3ec01e420 con 0x7fb3f80f5410 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.001+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8045ad0 con 0x7fb3f80f5410 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.001+0000 7fb3f57fa700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb3e802ff60 0x7fb3e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.001+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3ec041110 con 0x7fb3f80f5410 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.001+0000 7fb3f7fff700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb3e802ff60 0x7fb3e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.002+0000 7fb3f7fff700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb3e802ff60 0x7fb3e8032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3e40096a0 tx=0x7fb3e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:51.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.006+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3ec01cb10 con 0x7fb3f80f5410 2024-02-10T09:23:51.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.210+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3f810a3b0 con 0x7fb3f80f5410 2024-02-10T09:23:51.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.210+0000 7fb3f57fa700 1 -- 172.21.15.158:0/3125088750 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3ec004490 con 0x7fb3f80f5410 2024-02-10T09:23:51.212 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:51.212 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb3e802ff60 msgr2=0x7fb3e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb3e802ff60 0x7fb3e8032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fb3e40096a0 tx=0x7fb3e4006b40).stop 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 msgr2=0x7fb3f81094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f81094e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb3ec0089f0 tx=0x7fb3ec00b040).stop 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 shutdown_connections 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb3e802ff60 0x7fb3e8032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 --2- 172.21.15.158:0/3125088750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb3f80f5410 0x7fb3f81094e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:51.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 >> 172.21.15.158:0/3125088750 conn(0x7fb3f80f0b70 msgr2=0x7fb3f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:51.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 shutdown_connections 2024-02-10T09:23:51.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:51.212+0000 7fb3ff01d700 1 -- 172.21.15.158:0/3125088750 wait complete. 2024-02-10T09:23:51.214 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:52.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:50.048158+0000 mgr.smithi083.dowdym (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:52.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:51.211772+0000 mon.smithi083 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.158:0/3125088750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:52.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:52.838 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:54.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:52.048509+0000 mgr.smithi083.dowdym (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:54.678 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 -- 172.21.15.158:0/3306722406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c00f5410 msgr2=0x7f17c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 --2- 172.21.15.158:0/3306722406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c00f5410 0x7f17c00f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f17bc009fb0 tx=0x7f17bc009b30).stop 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 -- 172.21.15.158:0/3306722406 shutdown_connections 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 --2- 172.21.15.158:0/3306722406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c00f5410 0x7f17c00f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 -- 172.21.15.158:0/3306722406 >> 172.21.15.158:0/3306722406 conn(0x7f17c00f0b70 msgr2=0x7f17c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 -- 172.21.15.158:0/3306722406 shutdown_connections 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.676+0000 7f17c786a700 1 -- 172.21.15.158:0/3306722406 wait complete. 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c786a700 1 Processor -- start 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c786a700 1 -- start start 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c786a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 0x7f17c0109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c786a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0109b30 con 0x7f17c0060200 2024-02-10T09:23:54.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c5606700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 0x7f17c0109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:54.680 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c5606700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 0x7f17c0109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40540/0 (socket says 172.21.15.158:40540) 2024-02-10T09:23:54.680 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.677+0000 7f17c5606700 1 -- 172.21.15.158:0/942986390 learned_addr learned my addr 172.21.15.158:0/942986390 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:54.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17c5606700 1 -- 172.21.15.158:0/942986390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17bc005040 con 0x7f17c0060200 2024-02-10T09:23:54.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17c5606700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 0x7f17c0109640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f17c00f6510 tx=0x7f17bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:54.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17bc013380 con 0x7f17c0060200 2024-02-10T09:23:54.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c0109c70 con 0x7f17c0060200 2024-02-10T09:23:54.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17bc013790 con 0x7f17c0060200 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17bc01ee70 con 0x7f17c0060200 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.678+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c010a0a0 con 0x7f17c0060200 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.679+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17bc025060 con 0x7f17c0060200 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.679+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c0045ad0 con 0x7f17c0060200 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.679+0000 7f17b5ffb700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17ac02ff60 0x7f17ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.679+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17bc040e10 con 0x7f17c0060200 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.680+0000 7f17c4e05700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17ac02ff60 0x7f17ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:54.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.680+0000 7f17c4e05700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17ac02ff60 0x7f17ac032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f17b00096a0 tx=0x7f17b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:54.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.684+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17bc00f280 con 0x7f17c0060200 2024-02-10T09:23:54.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.891+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17c00fafa0 con 0x7f17c0060200 2024-02-10T09:23:54.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.891+0000 7f17b5ffb700 1 -- 172.21.15.158:0/942986390 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17bc02d2d0 con 0x7f17c0060200 2024-02-10T09:23:54.893 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:54.893 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:54.894 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17ac02ff60 msgr2=0x7f17ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:54.894 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17ac02ff60 0x7f17ac032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f17b00096a0 tx=0x7f17b0006b40).stop 2024-02-10T09:23:54.894 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 msgr2=0x7f17c0109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:54.894 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 0x7f17c0109640 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f17c00f6510 tx=0x7f17bc009b30).stop 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 shutdown_connections 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17ac02ff60 0x7f17ac032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 --2- 172.21.15.158:0/942986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c0060200 0x7f17c0109640 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 >> 172.21.15.158:0/942986390 conn(0x7f17c00f0b70 msgr2=0x7f17c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 shutdown_connections 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:54.893+0000 7f17c786a700 1 -- 172.21.15.158:0/942986390 wait complete. 2024-02-10T09:23:54.895 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:56.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:54.048856+0000 mgr.smithi083.dowdym (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:56.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:54.892896+0000 mon.smithi083 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.158:0/942986390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:23:56.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:23:56.462 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:23:58.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:56.049188+0000 mgr.smithi083.dowdym (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 -- 172.21.15.158:0/1925978926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 msgr2=0x7f503c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 --2- 172.21.15.158:0/1925978926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c0f55c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f5038009fb0 tx=0x7f5038009b30).stop 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 -- 172.21.15.158:0/1925978926 shutdown_connections 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 --2- 172.21.15.158:0/1925978926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c0f55c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 -- 172.21.15.158:0/1925978926 >> 172.21.15.158:0/1925978926 conn(0x7f503c0f09a0 msgr2=0x7f503c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 -- 172.21.15.158:0/1925978926 shutdown_connections 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 -- 172.21.15.158:0/1925978926 wait complete. 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.264+0000 7f504353a700 1 Processor -- start 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f504353a700 1 -- start start 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f504353a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f504353a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c1059c0 con 0x7f503c0f51a0 2024-02-10T09:23:58.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5042538700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5042538700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40558/0 (socket says 172.21.15.158:40558) 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5042538700 1 -- 172.21.15.158:0/735726937 learned_addr learned my addr 172.21.15.158:0/735726937 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5042538700 1 -- 172.21.15.158:0/735726937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5038005040 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5042538700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c1054d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5038009830 tx=0x7f503800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5038009320 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5038013380 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.265+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c105bc0 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.266+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5038014420 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.266+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c106050 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.266+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f503801e420 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.266+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c045ad0 con 0x7f503c0f51a0 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.266+0000 7f5032ffd700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f502802ff60 0x7f5028032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:23:58.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.266+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5038040eb0 con 0x7f503c0f51a0 2024-02-10T09:23:58.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.267+0000 7f5041d37700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f502802ff60 0x7f5028032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:23:58.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.267+0000 7f5041d37700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f502802ff60 0x7f5028032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f502c0096a0 tx=0x7f502c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:23:58.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.271+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f503801cb10 con 0x7f503c0f51a0 2024-02-10T09:23:58.477 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.476+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f503c0fad30 con 0x7f503c0f51a0 2024-02-10T09:23:58.478 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.476+0000 7f5032ffd700 1 -- 172.21.15.158:0/735726937 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5038004490 con 0x7f503c0f51a0 2024-02-10T09:23:58.478 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:23:58.478 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:23:58.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f502802ff60 msgr2=0x7f5028032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f502802ff60 0x7f5028032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f502c0096a0 tx=0x7f502c006b40).stop 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 msgr2=0x7f503c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c1054d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5038009830 tx=0x7f503800b040).stop 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 shutdown_connections 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f502802ff60 0x7f5028032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 --2- 172.21.15.158:0/735726937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f503c0f51a0 0x7f503c1054d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 >> 172.21.15.158:0/735726937 conn(0x7f503c0f09a0 msgr2=0x7f503c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 shutdown_connections 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:23:58.478+0000 7f504353a700 1 -- 172.21.15.158:0/735726937 wait complete. 2024-02-10T09:23:58.480 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:23:59.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:23:58.477862+0000 mon.smithi083 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.158:0/735726937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:00.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:23:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:23:58.049439+0000 mgr.smithi083.dowdym (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:00.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:00.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 -- 172.21.15.158:0/1071195690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 msgr2=0x7fbe800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 --2- 172.21.15.158:0/1071195690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe800f6930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fbe74009fb0 tx=0x7fbe74009b30).stop 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 -- 172.21.15.158:0/1071195690 shutdown_connections 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 --2- 172.21.15.158:0/1071195690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe800f6930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 -- 172.21.15.158:0/1071195690 >> 172.21.15.158:0/1071195690 conn(0x7fbe800f1c70 msgr2=0x7fbe800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 -- 172.21.15.158:0/1071195690 shutdown_connections 2024-02-10T09:24:01.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.874+0000 7fbe8556c700 1 -- 172.21.15.158:0/1071195690 wait complete. 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.875+0000 7fbe8556c700 1 Processor -- start 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.875+0000 7fbe8556c700 1 -- start start 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.875+0000 7fbe8556c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.875+0000 7fbe8556c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe80106cd0 con 0x7fbe800f6510 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe7effd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe7effd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52610/0 (socket says 172.21.15.158:52610) 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe7effd700 1 -- 172.21.15.158:0/3695337494 learned_addr learned my addr 172.21.15.158:0/3695337494 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe7effd700 1 -- 172.21.15.158:0/3695337494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe74005040 con 0x7fbe800f6510 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe7effd700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe801067e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fbe740089f0 tx=0x7fbe7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe74009320 con 0x7fbe800f6510 2024-02-10T09:24:01.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe74013380 con 0x7fbe800f6510 2024-02-10T09:24:01.879 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe80106ed0 con 0x7fbe800f6510 2024-02-10T09:24:01.879 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe7401ed80 con 0x7fbe800f6510 2024-02-10T09:24:01.879 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.876+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe80107360 con 0x7fbe800f6510 2024-02-10T09:24:01.879 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.877+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe7401e420 con 0x7fbe800f6510 2024-02-10T09:24:01.880 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.877+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe80045ad0 con 0x7fbe800f6510 2024-02-10T09:24:01.880 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.877+0000 7fbe677fe700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe6802ffb0 0x7fbe68032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:01.880 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.877+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe74041180 con 0x7fbe800f6510 2024-02-10T09:24:01.880 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.877+0000 7fbe7e7fc700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe6802ffb0 0x7fbe68032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:01.880 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.878+0000 7fbe7e7fc700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe6802ffb0 0x7fbe68032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbe700096a0 tx=0x7fbe70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:01.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:01.882+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe7401cb10 con 0x7fbe800f6510 2024-02-10T09:24:02.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:00.049776+0000 mgr.smithi083.dowdym (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:02.088 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.086+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe80108210 con 0x7fbe800f6510 2024-02-10T09:24:02.088 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.087+0000 7fbe677fe700 1 -- 172.21.15.158:0/3695337494 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe7401cb10 con 0x7fbe800f6510 2024-02-10T09:24:02.089 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:02.089 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe6802ffb0 msgr2=0x7fbe68032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe6802ffb0 0x7fbe68032470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbe700096a0 tx=0x7fbe70006b40).stop 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 msgr2=0x7fbe801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe801067e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fbe740089f0 tx=0x7fbe7400b040).stop 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 shutdown_connections 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe6802ffb0 0x7fbe68032470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 --2- 172.21.15.158:0/3695337494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe800f6510 0x7fbe801067e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 >> 172.21.15.158:0/3695337494 conn(0x7fbe800f1c70 msgr2=0x7fbe800f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:02.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 shutdown_connections 2024-02-10T09:24:02.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:02.089+0000 7fbe8556c700 1 -- 172.21.15.158:0/3695337494 wait complete. 2024-02-10T09:24:02.091 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:03.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:02 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:02.088616+0000 mon.smithi083 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.158:0/3695337494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:03.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:03.744 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:04.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:02.050055+0000 mgr.smithi083.dowdym (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:05.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 -- 172.21.15.158:0/2587115349 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 msgr2=0x7f54300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:05.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 --2- 172.21.15.158:0/2587115349 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54300f90f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f5424009fb0 tx=0x7f5424009b30).stop 2024-02-10T09:24:05.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 -- 172.21.15.158:0/2587115349 shutdown_connections 2024-02-10T09:24:05.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 --2- 172.21.15.158:0/2587115349 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54300f90f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:05.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 -- 172.21.15.158:0/2587115349 >> 172.21.15.158:0/2587115349 conn(0x7f54300f0b70 msgr2=0x7f54300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:05.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 -- 172.21.15.158:0/2587115349 shutdown_connections 2024-02-10T09:24:05.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 -- 172.21.15.158:0/2587115349 wait complete. 2024-02-10T09:24:05.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 Processor -- start 2024-02-10T09:24:05.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.683+0000 7f5436c76700 1 -- start start 2024-02-10T09:24:05.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5436c76700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54301067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5436c76700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5430106c90 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5434a12700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54301067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5434a12700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54301067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52620/0 (socket says 172.21.15.158:52620) 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5434a12700 1 -- 172.21.15.158:0/3371645692 learned_addr learned my addr 172.21.15.158:0/3371645692 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5434a12700 1 -- 172.21.15.158:0/3371645692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5424005040 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5434a12700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54301067a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5424009830 tx=0x7f542400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5424009320 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5430106e90 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5424013380 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.684+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5424014420 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.685+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5430107320 con 0x7f5430060280 2024-02-10T09:24:05.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.685+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5424014580 con 0x7f5430060280 2024-02-10T09:24:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.685+0000 7f542d7fa700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.686+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5424040eb0 con 0x7f5430060280 2024-02-10T09:24:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.686+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5430045ad0 con 0x7f5430060280 2024-02-10T09:24:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.686+0000 7f542ffff700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.686+0000 7f542ffff700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f542002ff60 0x7f5420032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f541c0096a0 tx=0x7f541c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:05.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.690+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54240273f0 con 0x7f5430060280 2024-02-10T09:24:05.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.895+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54300fc090 con 0x7f5430060280 2024-02-10T09:24:05.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.896+0000 7f542d7fa700 1 -- 172.21.15.158:0/3371645692 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f542401a230 con 0x7f5430060280 2024-02-10T09:24:05.897 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:05.898 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f542002ff60 msgr2=0x7f5420032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f542002ff60 0x7f5420032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f541c0096a0 tx=0x7f541c006b40).stop 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 msgr2=0x7f54301067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54301067a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5424009830 tx=0x7f542400b040).stop 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 shutdown_connections 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f542002ff60 0x7f5420032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 --2- 172.21.15.158:0/3371645692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5430060280 0x7f54301067a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:05.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 >> 172.21.15.158:0/3371645692 conn(0x7f54300f0b70 msgr2=0x7f54300f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:05.900 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 shutdown_connections 2024-02-10T09:24:05.900 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:05.898+0000 7f5436c76700 1 -- 172.21.15.158:0/3371645692 wait complete. 2024-02-10T09:24:05.900 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:06.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:04.050418+0000 mgr.smithi083.dowdym (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:07.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:06 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:05.897571+0000 mon.smithi083 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.158:0/3371645692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:07.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:07.511 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:08.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:06.050672+0000 mgr.smithi083.dowdym (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:09.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 -- 172.21.15.158:0/1750933931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 msgr2=0x7fb4400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:09.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 --2- 172.21.15.158:0/1750933931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4400f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb434009fb0 tx=0x7fb434009b30).stop 2024-02-10T09:24:09.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 -- 172.21.15.158:0/1750933931 shutdown_connections 2024-02-10T09:24:09.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 --2- 172.21.15.158:0/1750933931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4400f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:09.268 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 -- 172.21.15.158:0/1750933931 >> 172.21.15.158:0/1750933931 conn(0x7fb4400f0b70 msgr2=0x7fb4400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:09.268 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 -- 172.21.15.158:0/1750933931 shutdown_connections 2024-02-10T09:24:09.268 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 -- 172.21.15.158:0/1750933931 wait complete. 2024-02-10T09:24:09.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 Processor -- start 2024-02-10T09:24:09.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.266+0000 7fb448281700 1 -- start start 2024-02-10T09:24:09.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb448281700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4401094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb448281700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440060240 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb44601d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4401094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb44601d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4401094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52636/0 (socket says 172.21.15.158:52636) 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb44601d700 1 -- 172.21.15.158:0/3835907938 learned_addr learned my addr 172.21.15.158:0/3835907938 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb44601d700 1 -- 172.21.15.158:0/3835907938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb434005040 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb44601d700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4401094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb434005010 tx=0x7fb43400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb434009320 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb440060440 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb434013380 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.267+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb434014e50 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.268+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb440109b90 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.268+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb440045ad0 con 0x7fb4400f5410 2024-02-10T09:24:09.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.268+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4340095f0 con 0x7fb4400f5410 2024-02-10T09:24:09.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.268+0000 7fb432ffd700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb42c02ff10 0x7fb42c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:09.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.268+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb434040f70 con 0x7fb4400f5410 2024-02-10T09:24:09.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.269+0000 7fb44581c700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb42c02ff10 0x7fb42c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:09.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.269+0000 7fb44581c700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb42c02ff10 0x7fb42c0323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb43c0096a0 tx=0x7fb43c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:09.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.273+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb434014420 con 0x7fb4400f5410 2024-02-10T09:24:09.478 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.477+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb44010a3b0 con 0x7fb4400f5410 2024-02-10T09:24:09.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.478+0000 7fb432ffd700 1 -- 172.21.15.158:0/3835907938 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb43401d980 con 0x7fb4400f5410 2024-02-10T09:24:09.479 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:09.479 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:09.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb42c02ff10 msgr2=0x7fb42c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb42c02ff10 0x7fb42c0323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fb43c0096a0 tx=0x7fb43c006b40).stop 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 msgr2=0x7fb4401094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4401094e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fb434005010 tx=0x7fb43400b040).stop 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 shutdown_connections 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb42c02ff10 0x7fb42c0323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 --2- 172.21.15.158:0/3835907938 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb4400f5410 0x7fb4401094e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 >> 172.21.15.158:0/3835907938 conn(0x7fb4400f0b70 msgr2=0x7fb4400f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 shutdown_connections 2024-02-10T09:24:09.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:09.479+0000 7fb448281700 1 -- 172.21.15.158:0/3835907938 wait complete. 2024-02-10T09:24:09.482 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:10.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:08.050985+0000 mgr.smithi083.dowdym (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:10.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:09.479306+0000 mon.smithi083 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.158:0/3835907938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:11.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:11.082 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:12.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:10.051297+0000 mgr.smithi083.dowdym (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/2965998468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 msgr2=0x7fbcd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 --2- 172.21.15.158:0/2965998468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd40f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fbcc4009fb0 tx=0x7fbcc4009b30).stop 2024-02-10T09:24:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/2965998468 shutdown_connections 2024-02-10T09:24:12.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 --2- 172.21.15.158:0/2965998468 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd40f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/2965998468 >> 172.21.15.158:0/2965998468 conn(0x7fbcd40f0b70 msgr2=0x7fbcd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/2965998468 shutdown_connections 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.832+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/2965998468 wait complete. 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcdc0ad700 1 Processor -- start 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcdc0ad700 1 -- start start 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcdc0ad700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd4109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcdc0ad700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd4060220 con 0x7fbcd40f5410 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcd9e49700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd4109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcd9e49700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd4109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:60882/0 (socket says 172.21.15.158:60882) 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.833+0000 7fbcd9e49700 1 -- 172.21.15.158:0/3814573024 learned_addr learned my addr 172.21.15.158:0/3814573024 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcd9e49700 1 -- 172.21.15.158:0/3814573024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc4005040 con 0x7fbcd40f5410 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcd9e49700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd4109500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fbcc4005010 tx=0x7fbcc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc4009320 con 0x7fbcd40f5410 2024-02-10T09:24:12.836 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd4060420 con 0x7fbcd40f5410 2024-02-10T09:24:12.837 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcc4013380 con 0x7fbcd40f5410 2024-02-10T09:24:12.837 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc4014e50 con 0x7fbcd40f5410 2024-02-10T09:24:12.837 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.834+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd4109bb0 con 0x7fbcd40f5410 2024-02-10T09:24:12.838 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.835+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcc4014420 con 0x7fbcd40f5410 2024-02-10T09:24:12.838 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.835+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd4059100 con 0x7fbcd40f5410 2024-02-10T09:24:12.838 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.835+0000 7fbcca7fc700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcc002ff10 0x7fbcc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:12.838 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.835+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcc4040e30 con 0x7fbcd40f5410 2024-02-10T09:24:12.838 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.835+0000 7fbcd9648700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcc002ff10 0x7fbcc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:12.838 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.836+0000 7fbcd9648700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcc002ff10 0x7fbcc00323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fbcd00096a0 tx=0x7fbcd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:12.840 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:12.839+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcc40095f0 con 0x7fbcd40f5410 2024-02-10T09:24:13.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.047+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbcd4045ad0 con 0x7fbcd40f5410 2024-02-10T09:24:13.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcca7fc700 1 -- 172.21.15.158:0/3814573024 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbcc401db70 con 0x7fbcd40f5410 2024-02-10T09:24:13.050 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:13.050 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcc002ff10 msgr2=0x7fbcc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcc002ff10 0x7fbcc00323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fbcd00096a0 tx=0x7fbcd0006b40).stop 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 msgr2=0x7fbcd4109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd4109500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fbcc4005010 tx=0x7fbcc400b040).stop 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 shutdown_connections 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbcc002ff10 0x7fbcc00323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:13.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.049+0000 7fbcdc0ad700 1 --2- 172.21.15.158:0/3814573024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbcd40f5410 0x7fbcd4109500 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:13.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.050+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 >> 172.21.15.158:0/3814573024 conn(0x7fbcd40f0b70 msgr2=0x7fbcd40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:13.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.050+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 shutdown_connections 2024-02-10T09:24:13.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:13.050+0000 7fbcdc0ad700 1 -- 172.21.15.158:0/3814573024 wait complete. 2024-02-10T09:24:13.052 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:14.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:12.051572+0000 mgr.smithi083.dowdym (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:14.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:13.049373+0000 mon.smithi083 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.158:0/3814573024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:14.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:14.644 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:15.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:14.166789+0000 mon.smithi083 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:24:15.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:14.175456+0000 mon.smithi083 (mon.0) 143 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:24:16.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:14.051892+0000 mgr.smithi083.dowdym (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 -- 172.21.15.158:0/1599375219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 msgr2=0x7fca440f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:16.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 --2- 172.21.15.158:0/1599375219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca440f4be0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fca34009fb0 tx=0x7fca34009b30).stop 2024-02-10T09:24:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 -- 172.21.15.158:0/1599375219 shutdown_connections 2024-02-10T09:24:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 --2- 172.21.15.158:0/1599375219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca440f4be0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 -- 172.21.15.158:0/1599375219 >> 172.21.15.158:0/1599375219 conn(0x7fca440f02e0 msgr2=0x7fca440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 -- 172.21.15.158:0/1599375219 shutdown_connections 2024-02-10T09:24:16.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.370+0000 7fca494e1700 1 -- 172.21.15.158:0/1599375219 wait complete. 2024-02-10T09:24:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca494e1700 1 Processor -- start 2024-02-10T09:24:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca494e1700 1 -- start start 2024-02-10T09:24:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca494e1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca441067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:16.373 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca494e1700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca44106cc0 con 0x7fca440f47c0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca42ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca441067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca42ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca441067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:60896/0 (socket says 172.21.15.158:60896) 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca42ffd700 1 -- 172.21.15.158:0/2329530815 learned_addr learned my addr 172.21.15.158:0/2329530815 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.371+0000 7fca42ffd700 1 -- 172.21.15.158:0/2329530815 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34005040 con 0x7fca440f47c0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.372+0000 7fca42ffd700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca441067d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fca340089f0 tx=0x7fca3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.372+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca34009320 con 0x7fca440f47c0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.372+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca44106ec0 con 0x7fca440f47c0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.372+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca34013380 con 0x7fca440f47c0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.372+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3401ed80 con 0x7fca440f47c0 2024-02-10T09:24:16.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.372+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca44107350 con 0x7fca440f47c0 2024-02-10T09:24:16.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.373+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca34014420 con 0x7fca440f47c0 2024-02-10T09:24:16.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.373+0000 7fca2b7fe700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fca2c02ff60 0x7fca2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:16.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.373+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca44045ad0 con 0x7fca440f47c0 2024-02-10T09:24:16.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.373+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca340411c0 con 0x7fca440f47c0 2024-02-10T09:24:16.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.373+0000 7fca427fc700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fca2c02ff60 0x7fca2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:16.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.373+0000 7fca427fc700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fca2c02ff60 0x7fca2c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fca380096a0 tx=0x7fca38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:16.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.381+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca3401c500 con 0x7fca440f47c0 2024-02-10T09:24:16.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.586+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca44108200 con 0x7fca440f47c0 2024-02-10T09:24:16.588 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.586+0000 7fca2b7fe700 1 -- 172.21.15.158:0/2329530815 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca3401e420 con 0x7fca440f47c0 2024-02-10T09:24:16.588 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:16.588 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:16.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fca2c02ff60 msgr2=0x7fca2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:16.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fca2c02ff60 0x7fca2c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fca380096a0 tx=0x7fca38006b40).stop 2024-02-10T09:24:16.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 msgr2=0x7fca441067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:16.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca441067d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fca340089f0 tx=0x7fca3400b040).stop 2024-02-10T09:24:16.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 shutdown_connections 2024-02-10T09:24:16.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fca2c02ff60 0x7fca2c032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:16.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 --2- 172.21.15.158:0/2329530815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fca440f47c0 0x7fca441067d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:16.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 >> 172.21.15.158:0/2329530815 conn(0x7fca440f02e0 msgr2=0x7fca440f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:16.592 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 shutdown_connections 2024-02-10T09:24:16.592 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:16.588+0000 7fca494e1700 1 -- 172.21.15.158:0/2329530815 wait complete. 2024-02-10T09:24:16.592 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:17.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:16 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:16.587878+0000 mon.smithi083 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.158:0/2329530815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:18.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:16.052234+0000 mgr.smithi083.dowdym (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:18.159 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:18.159 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:19.910 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 -- 172.21.15.158:0/4135942174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 msgr2=0x7f30880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 --2- 172.21.15.158:0/4135942174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f30880f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f307c009fb0 tx=0x7f307c009b30).stop 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 -- 172.21.15.158:0/4135942174 shutdown_connections 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 --2- 172.21.15.158:0/4135942174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f30880f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 -- 172.21.15.158:0/4135942174 >> 172.21.15.158:0/4135942174 conn(0x7f30880f0b70 msgr2=0x7f30880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 -- 172.21.15.158:0/4135942174 shutdown_connections 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 -- 172.21.15.158:0/4135942174 wait complete. 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 Processor -- start 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.905+0000 7f308ed57700 1 -- start start 2024-02-10T09:24:19.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308ed57700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f3088105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308ed57700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3088105c10 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308caf3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f3088105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308caf3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f3088105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:37018/0 (socket says 172.21.15.158:37018) 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308caf3700 1 -- 172.21.15.158:0/3647458493 learned_addr learned my addr 172.21.15.158:0/3647458493 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308caf3700 1 -- 172.21.15.158:0/3647458493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c005040 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308caf3700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f3088105720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f307c005010 tx=0x7f307c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307c009320 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3088105e10 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f307c013380 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307c014e50 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.906+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30881062a0 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.907+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f307c0095f0 con 0x7f30880f5410 2024-02-10T09:24:19.912 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.907+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3088045ad0 con 0x7f30880f5410 2024-02-10T09:24:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.907+0000 7f30857fa700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f307802ff10 0x7f30780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.907+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f307c040e90 con 0x7f30880f5410 2024-02-10T09:24:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.908+0000 7f3087fff700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f307802ff10 0x7f30780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.908+0000 7f3087fff700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f307802ff10 0x7f30780323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f30740096a0 tx=0x7f3074006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:19.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:19.912+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f307c014420 con 0x7f30880f5410 2024-02-10T09:24:20.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:18.052549+0000 mgr.smithi083.dowdym (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:20.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.117+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3088107150 con 0x7f30880f5410 2024-02-10T09:24:20.119 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.118+0000 7f30857fa700 1 -- 172.21.15.158:0/3647458493 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f307c010230 con 0x7f30880f5410 2024-02-10T09:24:20.119 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:20.119 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:20.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f307802ff10 msgr2=0x7f30780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:20.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f307802ff10 0x7f30780323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f30740096a0 tx=0x7f3074006b40).stop 2024-02-10T09:24:20.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 msgr2=0x7f3088105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:20.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f3088105720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f307c005010 tx=0x7f307c00b040).stop 2024-02-10T09:24:20.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 shutdown_connections 2024-02-10T09:24:20.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f307802ff10 0x7f30780323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:20.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 --2- 172.21.15.158:0/3647458493 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30880f5410 0x7f3088105720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:20.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.119+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 >> 172.21.15.158:0/3647458493 conn(0x7f30880f0b70 msgr2=0x7f30880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:20.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.120+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 shutdown_connections 2024-02-10T09:24:20.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:20.120+0000 7f308ed57700 1 -- 172.21.15.158:0/3647458493 wait complete. 2024-02-10T09:24:20.123 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:21.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:20 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:20.119195+0000 mon.smithi083 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.158:0/3647458493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:21.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:21.767 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:22.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:20.052839+0000 mgr.smithi083.dowdym (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:23.620 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.586+0000 7fde634d1700 1 -- 172.21.15.158:0/3144119225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 msgr2=0x7fde5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.586+0000 7fde634d1700 1 --2- 172.21.15.158:0/3144119225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c0f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fde4c009fb0 tx=0x7fde4c009b30).stop 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.586+0000 7fde634d1700 1 -- 172.21.15.158:0/3144119225 shutdown_connections 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.586+0000 7fde634d1700 1 --2- 172.21.15.158:0/3144119225 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c0f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.586+0000 7fde634d1700 1 -- 172.21.15.158:0/3144119225 >> 172.21.15.158:0/3144119225 conn(0x7fde5c0f0b70 msgr2=0x7fde5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.587+0000 7fde634d1700 1 -- 172.21.15.158:0/3144119225 shutdown_connections 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.587+0000 7fde634d1700 1 -- 172.21.15.158:0/3144119225 wait complete. 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.587+0000 7fde634d1700 1 Processor -- start 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.587+0000 7fde634d1700 1 -- start start 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.587+0000 7fde634d1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.587+0000 7fde634d1700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c106ce0 con 0x7fde5c0f5410 2024-02-10T09:24:23.621 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde6126d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde6126d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:37038/0 (socket says 172.21.15.158:37038) 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde6126d700 1 -- 172.21.15.158:0/7090748 learned_addr learned my addr 172.21.15.158:0/7090748 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde6126d700 1 -- 172.21.15.158:0/7090748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c005040 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde6126d700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c1067f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fde4c005010 tx=0x7fde4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde4c009320 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c106ee0 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde4c013380 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde4c014e50 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.588+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c107370 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.589+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde4c0095f0 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.589+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5c045ad0 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.589+0000 7fde51ffb700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fde4802ff10 0x7fde480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.589+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde4c040f10 con 0x7fde5c0f5410 2024-02-10T09:24:23.622 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.590+0000 7fde60a6c700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fde4802ff10 0x7fde480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:23.623 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.590+0000 7fde60a6c700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fde4802ff10 0x7fde480323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fde580096a0 tx=0x7fde58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:23.623 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.594+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde4c014420 con 0x7fde5c0f5410 2024-02-10T09:24:23.801 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.799+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde5c108220 con 0x7fde5c0f5410 2024-02-10T09:24:23.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.800+0000 7fde51ffb700 1 -- 172.21.15.158:0/7090748 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fde4c010230 con 0x7fde5c0f5410 2024-02-10T09:24:23.802 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:23.802 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:23.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fde4802ff10 msgr2=0x7fde480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:23.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fde4802ff10 0x7fde480323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fde580096a0 tx=0x7fde58006b40).stop 2024-02-10T09:24:23.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 msgr2=0x7fde5c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:23.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c1067f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fde4c005010 tx=0x7fde4c00b040).stop 2024-02-10T09:24:23.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 shutdown_connections 2024-02-10T09:24:23.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fde4802ff10 0x7fde480323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:23.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 --2- 172.21.15.158:0/7090748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde5c0f5410 0x7fde5c1067f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:23.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 >> 172.21.15.158:0/7090748 conn(0x7fde5c0f0b70 msgr2=0x7fde5c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:23.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 shutdown_connections 2024-02-10T09:24:23.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:23.802+0000 7fde634d1700 1 -- 172.21.15.158:0/7090748 wait complete. 2024-02-10T09:24:23.804 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:24.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:22.053084+0000 mgr.smithi083.dowdym (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:23.801607+0000 mon.smithi083 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.158:0/7090748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:25.441 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:25.442 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:26.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:24.053351+0000 mgr.smithi083.dowdym (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 -- 172.21.15.158:0/2141064279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 msgr2=0x7f4f300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 --2- 172.21.15.158:0/2141064279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f300f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f4f2c009fb0 tx=0x7f4f2c009b30).stop 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 -- 172.21.15.158:0/2141064279 shutdown_connections 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 --2- 172.21.15.158:0/2141064279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f300f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 -- 172.21.15.158:0/2141064279 >> 172.21.15.158:0/2141064279 conn(0x7f4f300f1c70 msgr2=0x7f4f300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 -- 172.21.15.158:0/2141064279 shutdown_connections 2024-02-10T09:24:27.329 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.327+0000 7f4f38643700 1 -- 172.21.15.158:0/2141064279 wait complete. 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f38643700 1 Processor -- start 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f38643700 1 -- start start 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f38643700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f301067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f38643700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f30106cd0 con 0x7f4f300f6510 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f363df700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f301067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f363df700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f301067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:37044/0 (socket says 172.21.15.158:37044) 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f363df700 1 -- 172.21.15.158:0/182180198 learned_addr learned my addr 172.21.15.158:0/182180198 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f363df700 1 -- 172.21.15.158:0/182180198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c005040 con 0x7f4f300f6510 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.328+0000 7f4f363df700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f301067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4f2c005010 tx=0x7f4f2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f2c009320 con 0x7f4f300f6510 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f2c013380 con 0x7f4f300f6510 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f30106ed0 con 0x7f4f300f6510 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f2c014e50 con 0x7f4f300f6510 2024-02-10T09:24:27.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f30107360 con 0x7f4f300f6510 2024-02-10T09:24:27.331 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f2c014420 con 0x7f4f300f6510 2024-02-10T09:24:27.331 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.329+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f30045ad0 con 0x7f4f300f6510 2024-02-10T09:24:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.330+0000 7f4f26ffd700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4f1c02ff10 0x7f4f1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.330+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f2c040e70 con 0x7f4f300f6510 2024-02-10T09:24:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.330+0000 7f4f35bde700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4f1c02ff10 0x7f4f1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:27.332 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.330+0000 7f4f35bde700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4f1c02ff10 0x7f4f1c0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4f200096a0 tx=0x7f4f20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:27.334 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.334+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f2c01db70 con 0x7f4f300f6510 2024-02-10T09:24:27.539 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.539+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f30108210 con 0x7f4f300f6510 2024-02-10T09:24:27.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.540+0000 7f4f26ffd700 1 -- 172.21.15.158:0/182180198 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f2c01db70 con 0x7f4f300f6510 2024-02-10T09:24:27.540 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:27.540 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:27.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4f1c02ff10 msgr2=0x7f4f1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4f1c02ff10 0x7f4f1c0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4f200096a0 tx=0x7f4f20006b40).stop 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 msgr2=0x7f4f301067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f301067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f4f2c005010 tx=0x7f4f2c00b040).stop 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 shutdown_connections 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f4f1c02ff10 0x7f4f1c0323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 --2- 172.21.15.158:0/182180198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f300f6510 0x7f4f301067e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 >> 172.21.15.158:0/182180198 conn(0x7f4f300f1c70 msgr2=0x7f4f300f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 shutdown_connections 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:27.541+0000 7f4f38643700 1 -- 172.21.15.158:0/182180198 wait complete. 2024-02-10T09:24:27.542 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:28.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:26.053611+0000 mgr.smithi083.dowdym (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:28.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:27.540082+0000 mon.smithi083 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.158:0/182180198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:29.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:29.174 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:30.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:28.053894+0000 mgr.smithi083.dowdym (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 -- 172.21.15.158:0/2909339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 msgr2=0x7f85000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 --2- 172.21.15.158:0/2909339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85000f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f84f0009fb0 tx=0x7f84f0009b30).stop 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 -- 172.21.15.158:0/2909339342 shutdown_connections 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 --2- 172.21.15.158:0/2909339342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85000f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 -- 172.21.15.158:0/2909339342 >> 172.21.15.158:0/2909339342 conn(0x7f85000f0b70 msgr2=0x7f85000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 -- 172.21.15.158:0/2909339342 shutdown_connections 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 -- 172.21.15.158:0/2909339342 wait complete. 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 Processor -- start 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.137+0000 7f85061d3700 1 -- start start 2024-02-10T09:24:31.139 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f85061d3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85001094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f85061d3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8500060240 con 0x7f85000f5410 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84ff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85001094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84ff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85001094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36782/0 (socket says 172.21.15.158:36782) 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84ff7fe700 1 -- 172.21.15.158:0/806133847 learned_addr learned my addr 172.21.15.158:0/806133847 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84ff7fe700 1 -- 172.21.15.158:0/806133847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f0005040 con 0x7f85000f5410 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84ff7fe700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85001094e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f84f0005010 tx=0x7f84f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f0009320 con 0x7f85000f5410 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8500060440 con 0x7f85000f5410 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84f0013380 con 0x7f85000f5410 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.138+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f0014e50 con 0x7f85000f5410 2024-02-10T09:24:31.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.139+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8500109b90 con 0x7f85000f5410 2024-02-10T09:24:31.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.139+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84f00095f0 con 0x7f85000f5410 2024-02-10T09:24:31.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.139+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8500045ad0 con 0x7f85000f5410 2024-02-10T09:24:31.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.140+0000 7f84e7fff700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84e802ff10 0x7f84e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:31.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.140+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84f0040e70 con 0x7f85000f5410 2024-02-10T09:24:31.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.140+0000 7f84feffd700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84e802ff10 0x7f84e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:31.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.140+0000 7f84feffd700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84e802ff10 0x7f84e80323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f84f40096a0 tx=0x7f84f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:31.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.144+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84f0014420 con 0x7f85000f5410 2024-02-10T09:24:31.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.354+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f850010a3b0 con 0x7f85000f5410 2024-02-10T09:24:31.355 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.354+0000 7f84e7fff700 1 -- 172.21.15.158:0/806133847 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84f0010230 con 0x7f85000f5410 2024-02-10T09:24:31.355 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:31.355 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:31.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84e802ff10 msgr2=0x7f84e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84e802ff10 0x7f84e80323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f84f40096a0 tx=0x7f84f4006b40).stop 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 msgr2=0x7f85001094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85001094e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f84f0005010 tx=0x7f84f000b040).stop 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 shutdown_connections 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84e802ff10 0x7f84e80323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 --2- 172.21.15.158:0/806133847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f85000f5410 0x7f85001094e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 >> 172.21.15.158:0/806133847 conn(0x7f85000f0b70 msgr2=0x7f85000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 shutdown_connections 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:31.356+0000 7f85061d3700 1 -- 172.21.15.158:0/806133847 wait complete. 2024-02-10T09:24:31.357 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:32.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:30.054202+0000 mgr.smithi083.dowdym (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:32.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:31.354970+0000 mon.smithi083 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.158:0/806133847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:32.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:32.946 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:34.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:32.054536+0000 mgr.smithi083.dowdym (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:34.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 -- 172.21.15.158:0/102842974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 msgr2=0x7f960c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:34.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 --2- 172.21.15.158:0/102842974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c0f9190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f95fc009fb0 tx=0x7f95fc009b30).stop 2024-02-10T09:24:34.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 -- 172.21.15.158:0/102842974 shutdown_connections 2024-02-10T09:24:34.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 --2- 172.21.15.158:0/102842974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c0f9190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:34.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 -- 172.21.15.158:0/102842974 >> 172.21.15.158:0/102842974 conn(0x7f960c0f0b70 msgr2=0x7f960c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 -- 172.21.15.158:0/102842974 shutdown_connections 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.668+0000 7f96132d1700 1 -- 172.21.15.158:0/102842974 wait complete. 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f96132d1700 1 Processor -- start 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f96132d1700 1 -- start start 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f96132d1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f96132d1700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c106cc0 con 0x7f960c0601e0 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f961106d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f961106d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36808/0 (socket says 172.21.15.158:36808) 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f961106d700 1 -- 172.21.15.158:0/3708143378 learned_addr learned my addr 172.21.15.158:0/3708143378 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.669+0000 7f961106d700 1 -- 172.21.15.158:0/3708143378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc005040 con 0x7f960c0601e0 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.670+0000 7f961106d700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c1067d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f95fc009830 tx=0x7f95fc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.670+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95fc018070 con 0x7f960c0601e0 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.670+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c106ec0 con 0x7f960c0601e0 2024-02-10T09:24:34.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.670+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95fc013410 con 0x7f960c0601e0 2024-02-10T09:24:34.672 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.670+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95fc01c3c0 con 0x7f960c0601e0 2024-02-10T09:24:34.672 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.670+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c107350 con 0x7f960c0601e0 2024-02-10T09:24:34.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.671+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95fc01c680 con 0x7f960c0601e0 2024-02-10T09:24:34.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.671+0000 7f9601ffb700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f95f802ff60 0x7f95f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:34.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.671+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95fc0414d0 con 0x7f960c0601e0 2024-02-10T09:24:34.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.671+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c045ad0 con 0x7f960c0601e0 2024-02-10T09:24:34.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.671+0000 7f961086c700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f95f802ff60 0x7f95f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:34.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.672+0000 7f961086c700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f95f802ff60 0x7f95f8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:34.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.676+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95fc01ca80 con 0x7f960c0601e0 2024-02-10T09:24:34.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.881+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f960c108090 con 0x7f960c0601e0 2024-02-10T09:24:34.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.882+0000 7f9601ffb700 1 -- 172.21.15.158:0/3708143378 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95fc01b9c0 con 0x7f960c0601e0 2024-02-10T09:24:34.883 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:34.883 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:34.884 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f95f802ff60 msgr2=0x7f95f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:34.884 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f95f802ff60 0x7f95f8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).stop 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 msgr2=0x7f960c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c1067d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f95fc009830 tx=0x7f95fc009b30).stop 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 shutdown_connections 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f95f802ff60 0x7f95f8032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 --2- 172.21.15.158:0/3708143378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f960c0601e0 0x7f960c1067d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 >> 172.21.15.158:0/3708143378 conn(0x7f960c0f0b70 msgr2=0x7f960c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 shutdown_connections 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:34.884+0000 7f96132d1700 1 -- 172.21.15.158:0/3708143378 wait complete. 2024-02-10T09:24:34.885 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:36.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:34.054926+0000 mgr.smithi083.dowdym (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:36.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:34.882458+0000 mon.smithi083 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.158:0/3708143378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:36.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:36.443 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:36.055209+0000 mgr.smithi083.dowdym (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:38.282 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 -- 172.21.15.158:0/4139939055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 msgr2=0x7f24b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 --2- 172.21.15.158:0/4139939055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b80f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f24a0009fb0 tx=0x7f24a0009b30).stop 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 -- 172.21.15.158:0/4139939055 shutdown_connections 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 --2- 172.21.15.158:0/4139939055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b80f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 -- 172.21.15.158:0/4139939055 >> 172.21.15.158:0/4139939055 conn(0x7f24b80f1c70 msgr2=0x7f24b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 -- 172.21.15.158:0/4139939055 shutdown_connections 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.281+0000 7f24bcf88700 1 -- 172.21.15.158:0/4139939055 wait complete. 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24bcf88700 1 Processor -- start 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24bcf88700 1 -- start start 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24bcf88700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:38.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24bcf88700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b8069e20 con 0x7f24b80f6510 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24b659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24b659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36828/0 (socket says 172.21.15.158:36828) 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.282+0000 7f24b659c700 1 -- 172.21.15.158:0/546020095 learned_addr learned my addr 172.21.15.158:0/546020095 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24b659c700 1 -- 172.21.15.158:0/546020095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a0005040 con 0x7f24b80f6510 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24b659c700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b806b9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f24a0005010 tx=0x7f24a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a0009320 con 0x7f24b80f6510 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b8069fc0 con 0x7f24b80f6510 2024-02-10T09:24:38.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24a0013380 con 0x7f24b80f6510 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a0014e50 con 0x7f24b80f6510 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.283+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b806a450 con 0x7f24b80f6510 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.284+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24a0014420 con 0x7f24b80f6510 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.284+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b8059100 con 0x7f24b80f6510 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.284+0000 7f24aeffd700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f24a402ff10 0x7f24a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.284+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24a0040e30 con 0x7f24b80f6510 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.284+0000 7f24b5d9b700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f24a402ff10 0x7f24a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:38.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.285+0000 7f24b5d9b700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f24a402ff10 0x7f24a40323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f24a80096a0 tx=0x7f24a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:38.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.288+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24a00095f0 con 0x7f24b80f6510 2024-02-10T09:24:38.492 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.492+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24b80fc0a0 con 0x7f24b80f6510 2024-02-10T09:24:38.493 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.492+0000 7f24aeffd700 1 -- 172.21.15.158:0/546020095 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f24a001db70 con 0x7f24b80f6510 2024-02-10T09:24:38.493 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:38.493 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:38.494 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f24a402ff10 msgr2=0x7f24a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:38.494 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f24a402ff10 0x7f24a40323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f24a80096a0 tx=0x7f24a8006b40).stop 2024-02-10T09:24:38.494 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 msgr2=0x7f24b806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:38.494 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b806b9a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f24a0005010 tx=0x7f24a000b040).stop 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 shutdown_connections 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f24a402ff10 0x7f24a40323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 --2- 172.21.15.158:0/546020095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b80f6510 0x7f24b806b9a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 >> 172.21.15.158:0/546020095 conn(0x7f24b80f1c70 msgr2=0x7f24b80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 shutdown_connections 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:38.494+0000 7f24bcf88700 1 -- 172.21.15.158:0/546020095 wait complete. 2024-02-10T09:24:38.495 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:39.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:38 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:38.492903+0000 mon.smithi083 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.158:0/546020095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:40.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:40.048 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:40.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:38.055493+0000 mgr.smithi083.dowdym (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 -- 172.21.15.158:0/2429625738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 msgr2=0x7fbd800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 --2- 172.21.15.158:0/2429625738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbd70009fb0 tx=0x7fbd70009b30).stop 2024-02-10T09:24:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 -- 172.21.15.158:0/2429625738 shutdown_connections 2024-02-10T09:24:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 --2- 172.21.15.158:0/2429625738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:41.946 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 -- 172.21.15.158:0/2429625738 >> 172.21.15.158:0/2429625738 conn(0x7fbd800f0b70 msgr2=0x7fbd800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 -- 172.21.15.158:0/2429625738 shutdown_connections 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.905+0000 7fbd87294700 1 -- 172.21.15.158:0/2429625738 wait complete. 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd87294700 1 Processor -- start 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd87294700 1 -- start start 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd87294700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd87294700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800fb960 con 0x7fbd800f5410 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd85030700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd85030700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39082/0 (socket says 172.21.15.158:39082) 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd85030700 1 -- 172.21.15.158:0/613317315 learned_addr learned my addr 172.21.15.158:0/613317315 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.906+0000 7fbd85030700 1 -- 172.21.15.158:0/613317315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd70005040 con 0x7fbd800f5410 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.907+0000 7fbd85030700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800fd4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fbd700089f0 tx=0x7fbd7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.907+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd70009320 con 0x7fbd800f5410 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.907+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd800fbb00 con 0x7fbd800f5410 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.907+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd800fbf90 con 0x7fbd800f5410 2024-02-10T09:24:41.947 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.907+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd70013380 con 0x7fbd800f5410 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.907+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd7001ed80 con 0x7fbd800f5410 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.908+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd7001e420 con 0x7fbd800f5410 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.908+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80059100 con 0x7fbd800f5410 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.908+0000 7fbd75ffb700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbd6c02ff60 0x7fbd6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.908+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd70041030 con 0x7fbd800f5410 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.908+0000 7fbd8482f700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbd6c02ff60 0x7fbd6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.909+0000 7fbd8482f700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbd6c02ff60 0x7fbd6c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fbd7c0096a0 tx=0x7fbd7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:41.948 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:41.913+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd700095f0 con 0x7fbd800f5410 2024-02-10T09:24:42.119 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.119+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd800fafa0 con 0x7fbd800f5410 2024-02-10T09:24:42.119 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.119+0000 7fbd75ffb700 1 -- 172.21.15.158:0/613317315 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd7001cb10 con 0x7fbd800f5410 2024-02-10T09:24:42.120 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:42.120 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:42.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbd6c02ff60 msgr2=0x7fbd6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:42.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbd6c02ff60 0x7fbd6c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fbd7c0096a0 tx=0x7fbd7c006b40).stop 2024-02-10T09:24:42.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 msgr2=0x7fbd800fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800fd4e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fbd700089f0 tx=0x7fbd7000b040).stop 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 shutdown_connections 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbd6c02ff60 0x7fbd6c032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 --2- 172.21.15.158:0/613317315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbd800f5410 0x7fbd800fd4e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 >> 172.21.15.158:0/613317315 conn(0x7fbd800f0b70 msgr2=0x7fbd8005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 shutdown_connections 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:42.121+0000 7fbd87294700 1 -- 172.21.15.158:0/613317315 wait complete. 2024-02-10T09:24:42.122 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:40.055841+0000 mgr.smithi083.dowdym (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:43.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:42 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:42.119905+0000 mon.smithi083 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.158:0/613317315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:43.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:43.726 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:44.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:42.056220+0000 mgr.smithi083.dowdym (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/3393693028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 msgr2=0x7fcaf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 --2- 172.21.15.158:0/3393693028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf00f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fcae4009fb0 tx=0x7fcae4009b30).stop 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/3393693028 shutdown_connections 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 --2- 172.21.15.158:0/3393693028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf00f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/3393693028 >> 172.21.15.158:0/3393693028 conn(0x7fcaf00f0b70 msgr2=0x7fcaf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/3393693028 shutdown_connections 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.443+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/3393693028 wait complete. 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.444+0000 7fcaf6f2b700 1 Processor -- start 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.444+0000 7fcaf6f2b700 1 -- start start 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.444+0000 7fcaf6f2b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf0109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.444+0000 7fcaf6f2b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0060220 con 0x7fcaf00f5410 2024-02-10T09:24:45.460 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf4cc7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf0109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf4cc7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf0109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39092/0 (socket says 172.21.15.158:39092) 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf4cc7700 1 -- 172.21.15.158:0/1687308091 learned_addr learned my addr 172.21.15.158:0/1687308091 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf4cc7700 1 -- 172.21.15.158:0/1687308091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4005040 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf4cc7700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf0109530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fcae4005010 tx=0x7fcae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcae4009320 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf0060420 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcae4013380 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcae4014e50 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.445+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf0109be0 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.446+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcae40095f0 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.446+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf0059100 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.446+0000 7fcaed7fa700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcae002ff10 0x7fcae00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.446+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcae4040e90 con 0x7fcaf00f5410 2024-02-10T09:24:45.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.447+0000 7fcaeffff700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcae002ff10 0x7fcae00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:45.462 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.447+0000 7fcaeffff700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcae002ff10 0x7fcae00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcadc0096a0 tx=0x7fcadc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:45.462 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.451+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcae4014420 con 0x7fcaf00f5410 2024-02-10T09:24:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.658+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaf0045ad0 con 0x7fcaf00f5410 2024-02-10T09:24:45.659 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.659+0000 7fcaed7fa700 1 -- 172.21.15.158:0/1687308091 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcae4010230 con 0x7fcaf00f5410 2024-02-10T09:24:45.660 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:45.660 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcae002ff10 msgr2=0x7fcae00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcae002ff10 0x7fcae00323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcadc0096a0 tx=0x7fcadc006b40).stop 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 msgr2=0x7fcaf0109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf0109530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fcae4005010 tx=0x7fcae400b040).stop 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 shutdown_connections 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcae002ff10 0x7fcae00323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:45.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 --2- 172.21.15.158:0/1687308091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcaf00f5410 0x7fcaf0109530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:45.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 >> 172.21.15.158:0/1687308091 conn(0x7fcaf00f0b70 msgr2=0x7fcaf00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:45.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 shutdown_connections 2024-02-10T09:24:45.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:45.661+0000 7fcaf6f2b700 1 -- 172.21.15.158:0/1687308091 wait complete. 2024-02-10T09:24:45.662 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:44.056559+0000 mgr.smithi083.dowdym (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:45.659620+0000 mon.smithi083 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.158:0/1687308091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:47.353 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:47.353 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:46.056889+0000 mgr.smithi083.dowdym (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:49.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 -- 172.21.15.158:0/3706686906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 msgr2=0x7f2ad80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:49.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 --2- 172.21.15.158:0/3706686906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad80f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2ac8009fb0 tx=0x7f2ac8009b30).stop 2024-02-10T09:24:49.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 -- 172.21.15.158:0/3706686906 shutdown_connections 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 --2- 172.21.15.158:0/3706686906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad80f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 -- 172.21.15.158:0/3706686906 >> 172.21.15.158:0/3706686906 conn(0x7f2ad80f1c70 msgr2=0x7f2ad80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 -- 172.21.15.158:0/3706686906 shutdown_connections 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.262+0000 7f2ae01da700 1 -- 172.21.15.158:0/3706686906 wait complete. 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2ae01da700 1 Processor -- start 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2ae01da700 1 -- start start 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2ae01da700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2ae01da700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad8106cd0 con 0x7f2ad80f6510 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2addf76700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2addf76700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39108/0 (socket says 172.21.15.158:39108) 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.263+0000 7f2addf76700 1 -- 172.21.15.158:0/1325909825 learned_addr learned my addr 172.21.15.158:0/1325909825 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2addf76700 1 -- 172.21.15.158:0/1325909825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8005040 con 0x7f2ad80f6510 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2addf76700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad81067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2ac8005010 tx=0x7f2ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:49.265 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac8009320 con 0x7f2ad80f6510 2024-02-10T09:24:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad8106ed0 con 0x7f2ad80f6510 2024-02-10T09:24:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2ac8013380 con 0x7f2ad80f6510 2024-02-10T09:24:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ac8014e50 con 0x7f2ad80f6510 2024-02-10T09:24:49.266 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.264+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad8107360 con 0x7f2ad80f6510 2024-02-10T09:24:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.265+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2ac8014420 con 0x7f2ad80f6510 2024-02-10T09:24:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.265+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad8045ad0 con 0x7f2ad80f6510 2024-02-10T09:24:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.265+0000 7f2ace7fc700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2ac402ff10 0x7f2ac40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.265+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2ac8040e30 con 0x7f2ad80f6510 2024-02-10T09:24:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.265+0000 7f2add775700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2ac402ff10 0x7f2ac40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:49.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.265+0000 7f2add775700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2ac402ff10 0x7f2ac40323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2ad40096a0 tx=0x7f2ad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:49.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.269+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2ac80095f0 con 0x7f2ad80f6510 2024-02-10T09:24:49.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.472+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ad8108210 con 0x7f2ad80f6510 2024-02-10T09:24:49.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.473+0000 7f2ace7fc700 1 -- 172.21.15.158:0/1325909825 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2ac801db70 con 0x7f2ad80f6510 2024-02-10T09:24:49.473 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:49.474 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:49.475 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2ac402ff10 msgr2=0x7f2ac40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:49.475 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2ac402ff10 0x7f2ac40323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2ad40096a0 tx=0x7f2ad4006b40).stop 2024-02-10T09:24:49.475 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 msgr2=0x7f2ad81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:49.475 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad81067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2ac8005010 tx=0x7f2ac800b040).stop 2024-02-10T09:24:49.475 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 shutdown_connections 2024-02-10T09:24:49.475 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2ac402ff10 0x7f2ac40323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:49.476 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 --2- 172.21.15.158:0/1325909825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ad80f6510 0x7f2ad81067e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:49.476 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 >> 172.21.15.158:0/1325909825 conn(0x7f2ad80f1c70 msgr2=0x7f2ad80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:49.476 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 shutdown_connections 2024-02-10T09:24:49.476 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:49.475+0000 7f2ae01da700 1 -- 172.21.15.158:0/1325909825 wait complete. 2024-02-10T09:24:49.476 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:50.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:48.057154+0000 mgr.smithi083.dowdym (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:49.473635+0000 mon.smithi083 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.158:0/1325909825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:51.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:51.161 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:52.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:50.057453+0000 mgr.smithi083.dowdym (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:53.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 -- 172.21.15.158:0/3617982315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 msgr2=0x7f18dc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:53.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 --2- 172.21.15.158:0/3617982315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc0f66c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f18cc009fb0 tx=0x7f18cc009b30).stop 2024-02-10T09:24:53.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 -- 172.21.15.158:0/3617982315 shutdown_connections 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 --2- 172.21.15.158:0/3617982315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc0f66c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 -- 172.21.15.158:0/3617982315 >> 172.21.15.158:0/3617982315 conn(0x7f18dc0f1ac0 msgr2=0x7f18dc0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 -- 172.21.15.158:0/3617982315 shutdown_connections 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 -- 172.21.15.158:0/3617982315 wait complete. 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 Processor -- start 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.045+0000 7f18db59e700 1 -- start start 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18db59e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:53.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18db59e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc106ab0 con 0x7f18dc0f62a0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18da59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18da59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57426/0 (socket says 172.21.15.158:57426) 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18da59c700 1 -- 172.21.15.158:0/1191190357 learned_addr learned my addr 172.21.15.158:0/1191190357 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18da59c700 1 -- 172.21.15.158:0/1191190357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc005040 con 0x7f18dc0f62a0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18da59c700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc1065c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f18cc0089f0 tx=0x7f18cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18cc009320 con 0x7f18dc0f62a0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc106cb0 con 0x7f18dc0f62a0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18cc013380 con 0x7f18dc0f62a0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18cc014e20 con 0x7f18dc0f62a0 2024-02-10T09:24:53.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.046+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc107140 con 0x7f18dc0f62a0 2024-02-10T09:24:53.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.047+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc045ad0 con 0x7f18dc0f62a0 2024-02-10T09:24:53.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.047+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18cc0095f0 con 0x7f18dc0f62a0 2024-02-10T09:24:53.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.047+0000 7f18d2ffd700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f18c802ff10 0x7f18c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:53.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.047+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18cc040f10 con 0x7f18dc0f62a0 2024-02-10T09:24:53.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.048+0000 7f18d9d9b700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f18c802ff10 0x7f18c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:53.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.048+0000 7f18d9d9b700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f18c802ff10 0x7f18c80323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f18c40096a0 tx=0x7f18c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:53.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.052+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18cc014420 con 0x7f18dc0f62a0 2024-02-10T09:24:53.255 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.254+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18dc107ff0 con 0x7f18dc0f62a0 2024-02-10T09:24:53.255 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.255+0000 7f18d2ffd700 1 -- 172.21.15.158:0/1191190357 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18cc01d980 con 0x7f18dc0f62a0 2024-02-10T09:24:53.255 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:53.256 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f18c802ff10 msgr2=0x7f18c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f18c802ff10 0x7f18c80323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f18c40096a0 tx=0x7f18c4006b40).stop 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 msgr2=0x7f18dc1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc1065c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f18cc0089f0 tx=0x7f18cc00b040).stop 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 shutdown_connections 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f18c802ff10 0x7f18c80323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 --2- 172.21.15.158:0/1191190357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18dc0f62a0 0x7f18dc1065c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:53.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 >> 172.21.15.158:0/1191190357 conn(0x7f18dc0f1ac0 msgr2=0x7f18dc0f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:53.258 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 shutdown_connections 2024-02-10T09:24:53.258 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:53.257+0000 7f18db59e700 1 -- 172.21.15.158:0/1191190357 wait complete. 2024-02-10T09:24:53.258 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:54.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:52.057719+0000 mgr.smithi083.dowdym (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:54.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:53.255537+0000 mon.smithi083 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.158:0/1191190357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:54.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:54.830 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:24:56.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:54.058065+0000 mgr.smithi083.dowdym (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 -- 172.21.15.158:0/2186819172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 msgr2=0x7f0de00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 --2- 172.21.15.158:0/2186819172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de00f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f0dd0009fb0 tx=0x7f0dd0009b30).stop 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 -- 172.21.15.158:0/2186819172 shutdown_connections 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 --2- 172.21.15.158:0/2186819172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de00f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 -- 172.21.15.158:0/2186819172 >> 172.21.15.158:0/2186819172 conn(0x7f0de00f0b70 msgr2=0x7f0de00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 -- 172.21.15.158:0/2186819172 shutdown_connections 2024-02-10T09:24:56.684 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.662+0000 7f0de6997700 1 -- 172.21.15.158:0/2186819172 wait complete. 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0de6997700 1 Processor -- start 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0de6997700 1 -- start start 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0de6997700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0de6997700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de0105c10 con 0x7f0de00f5410 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0ddffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0ddffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57438/0 (socket says 172.21.15.158:57438) 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.663+0000 7f0ddffff700 1 -- 172.21.15.158:0/1138435071 learned_addr learned my addr 172.21.15.158:0/1138435071 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0ddffff700 1 -- 172.21.15.158:0/1138435071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd0005040 con 0x7f0de00f5410 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0ddffff700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de0105720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f0dd0005010 tx=0x7f0dd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:56.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dd0009320 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de0105e10 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dd0013380 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dd0014e50 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.664+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de01062a0 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.665+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0dd00263a0 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.665+0000 7f0ddcff9700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dcc02ff10 0x7f0dcc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.665+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0dd0041160 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.665+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de0045ad0 con 0x7f0de00f5410 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.665+0000 7f0ddf7fe700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dcc02ff10 0x7f0dcc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.665+0000 7f0ddf7fe700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dcc02ff10 0x7f0dcc0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0dc80096a0 tx=0x7f0dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:24:56.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.670+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0dd0012920 con 0x7f0de00f5410 2024-02-10T09:24:56.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.877+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0de0106fe0 con 0x7f0de00f5410 2024-02-10T09:24:56.879 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.878+0000 7f0ddcff9700 1 -- 172.21.15.158:0/1138435071 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0dd002b030 con 0x7f0de00f5410 2024-02-10T09:24:56.879 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:24:56.879 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dcc02ff10 msgr2=0x7f0dcc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dcc02ff10 0x7f0dcc0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f0dc80096a0 tx=0x7f0dc8006b40).stop 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 msgr2=0x7f0de0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de0105720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f0dd0005010 tx=0x7f0dd000b040).stop 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 shutdown_connections 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dcc02ff10 0x7f0dcc0323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 --2- 172.21.15.158:0/1138435071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0de00f5410 0x7f0de0105720 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 >> 172.21.15.158:0/1138435071 conn(0x7f0de00f0b70 msgr2=0x7f0de00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 shutdown_connections 2024-02-10T09:24:56.881 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:24:56.879+0000 7f0de6997700 1 -- 172.21.15.158:0/1138435071 wait complete. 2024-02-10T09:24:56.882 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:24:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:56.058347+0000 mgr.smithi083.dowdym (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:24:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:24:56.879182+0000 mon.smithi083 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.158:0/1138435071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:24:58.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:24:58.572 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:00.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:24:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:24:58.058681+0000 mgr.smithi083.dowdym (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/223517927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 msgr2=0x7f6adc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 --2- 172.21.15.158:0/223517927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc0f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6ac4009fb0 tx=0x7f6ac4009b30).stop 2024-02-10T09:25:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/223517927 shutdown_connections 2024-02-10T09:25:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 --2- 172.21.15.158:0/223517927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc0f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/223517927 >> 172.21.15.158:0/223517927 conn(0x7f6adc0f0b70 msgr2=0x7f6adc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:00.537 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/223517927 shutdown_connections 2024-02-10T09:25:00.538 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.535+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/223517927 wait complete. 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.536+0000 7f6ae1c3e700 1 Processor -- start 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.536+0000 7f6ae1c3e700 1 -- start start 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.536+0000 7f6ae1c3e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc1089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.536+0000 7f6ae1c3e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6adc108ee0 con 0x7f6adc0f5410 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6adb7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc1089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6adb7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc1089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49102/0 (socket says 172.21.15.158:49102) 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6adb7fe700 1 -- 172.21.15.158:0/4156545604 learned_addr learned my addr 172.21.15.158:0/4156545604 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6adb7fe700 1 -- 172.21.15.158:0/4156545604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac4005040 con 0x7f6adc0f5410 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6adb7fe700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc1089f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6ac4005010 tx=0x7f6ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ac4009320 con 0x7f6adc0f5410 2024-02-10T09:25:00.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6adc1090e0 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ac4013380 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ac4014e50 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.537+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6adc109570 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.538+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ac40095f0 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.538+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6adc045ad0 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.538+0000 7f6ad3fff700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6ac802ff10 0x7f6ac80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.538+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ac4040fa0 con 0x7f6adc0f5410 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.539+0000 7f6adaffd700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6ac802ff10 0x7f6ac80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:00.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.539+0000 7f6adaffd700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6ac802ff10 0x7f6ac80323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6acc0096a0 tx=0x7f6acc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:00.544 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.543+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ac4014420 con 0x7f6adc0f5410 2024-02-10T09:25:00.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.747+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6adc0fafa0 con 0x7f6adc0f5410 2024-02-10T09:25:00.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.748+0000 7f6ad3fff700 1 -- 172.21.15.158:0/4156545604 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ac4010230 con 0x7f6adc0f5410 2024-02-10T09:25:00.749 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:00.750 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.749+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6ac802ff10 msgr2=0x7f6ac80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.749+0000 7f6ae1c3e700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6ac802ff10 0x7f6ac80323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6acc0096a0 tx=0x7f6acc006b40).stop 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.749+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 msgr2=0x7f6adc1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.749+0000 7f6ae1c3e700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc1089f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6ac4005010 tx=0x7f6ac400b040).stop 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.750+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 shutdown_connections 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.750+0000 7f6ae1c3e700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6ac802ff10 0x7f6ac80323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.750+0000 7f6ae1c3e700 1 --2- 172.21.15.158:0/4156545604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6adc0f5410 0x7f6adc1089f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.750+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 >> 172.21.15.158:0/4156545604 conn(0x7f6adc0f0b70 msgr2=0x7f6adc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.750+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 shutdown_connections 2024-02-10T09:25:00.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:00.750+0000 7f6ae1c3e700 1 -- 172.21.15.158:0/4156545604 wait complete. 2024-02-10T09:25:00.752 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:01.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:00 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:00.749615+0000 mon.smithi083 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.158:0/4156545604' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:02.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:00.059006+0000 mgr.smithi083.dowdym (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:02.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:02.349 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:04.256 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 -- 172.21.15.158:0/3437354891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 msgr2=0x7f58640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:04.256 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 --2- 172.21.15.158:0/3437354891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58640f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f5854009fb0 tx=0x7f5854009b30).stop 2024-02-10T09:25:04.256 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 -- 172.21.15.158:0/3437354891 shutdown_connections 2024-02-10T09:25:04.256 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 --2- 172.21.15.158:0/3437354891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58640f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 -- 172.21.15.158:0/3437354891 >> 172.21.15.158:0/3437354891 conn(0x7f58640f0b70 msgr2=0x7f58640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 -- 172.21.15.158:0/3437354891 shutdown_connections 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.253+0000 7f586b564700 1 -- 172.21.15.158:0/3437354891 wait complete. 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f586b564700 1 Processor -- start 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f586b564700 1 -- start start 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f586b564700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58641067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f586b564700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5864106cc0 con 0x7f58640f5410 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f5869300700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58641067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f5869300700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58641067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49128/0 (socket says 172.21.15.158:49128) 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f5869300700 1 -- 172.21.15.158:0/1911919791 learned_addr learned my addr 172.21.15.158:0/1911919791 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.254+0000 7f5869300700 1 -- 172.21.15.158:0/1911919791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5854005040 con 0x7f58640f5410 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.255+0000 7f5869300700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58641067d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5854005010 tx=0x7f585400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.255+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5854009320 con 0x7f58640f5410 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.255+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5864106ec0 con 0x7f58640f5410 2024-02-10T09:25:04.257 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.255+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5854013380 con 0x7f58640f5410 2024-02-10T09:25:04.258 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.255+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5854014e50 con 0x7f58640f5410 2024-02-10T09:25:04.258 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.255+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5864107350 con 0x7f58640f5410 2024-02-10T09:25:04.258 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.256+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58540263a0 con 0x7f58640f5410 2024-02-10T09:25:04.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.256+0000 7f5859ffb700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f585002ff10 0x7f58500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:04.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.256+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5854040fa0 con 0x7f58640f5410 2024-02-10T09:25:04.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.256+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5864045ad0 con 0x7f58640f5410 2024-02-10T09:25:04.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.256+0000 7f5868aff700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f585002ff10 0x7f58500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:04.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.257+0000 7f5868aff700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f585002ff10 0x7f58500323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f58600096a0 tx=0x7f5860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:04.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.261+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5854012920 con 0x7f58640f5410 2024-02-10T09:25:04.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:02.059355+0000 mgr.smithi083.dowdym (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:04.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.466+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5864108070 con 0x7f58640f5410 2024-02-10T09:25:04.468 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.466+0000 7f5859ffb700 1 -- 172.21.15.158:0/1911919791 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f585402b020 con 0x7f58640f5410 2024-02-10T09:25:04.468 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:04.468 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f585002ff10 msgr2=0x7f58500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f585002ff10 0x7f58500323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f58600096a0 tx=0x7f5860006b40).stop 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 msgr2=0x7f58641067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58641067d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f5854005010 tx=0x7f585400b040).stop 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 shutdown_connections 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f585002ff10 0x7f58500323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 --2- 172.21.15.158:0/1911919791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58640f5410 0x7f58641067d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 >> 172.21.15.158:0/1911919791 conn(0x7f58640f0b70 msgr2=0x7f586405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 shutdown_connections 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:04.468+0000 7f586b564700 1 -- 172.21.15.158:0/1911919791 wait complete. 2024-02-10T09:25:04.470 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:05.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:04.467877+0000 mon.smithi083 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.158:0/1911919791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:06.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:06.131 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:06.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:04.059668+0000 mgr.smithi083.dowdym (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 -- 172.21.15.158:0/1543914189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 msgr2=0x7f7b640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 --2- 172.21.15.158:0/1543914189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b640f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7b58009fb0 tx=0x7f7b58009b30).stop 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 -- 172.21.15.158:0/1543914189 shutdown_connections 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 --2- 172.21.15.158:0/1543914189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b640f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 -- 172.21.15.158:0/1543914189 >> 172.21.15.158:0/1543914189 conn(0x7f7b640f0b70 msgr2=0x7f7b640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 -- 172.21.15.158:0/1543914189 shutdown_connections 2024-02-10T09:25:07.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.945+0000 7f7b69445700 1 -- 172.21.15.158:0/1543914189 wait complete. 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b69445700 1 Processor -- start 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b69445700 1 -- start start 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b69445700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b64109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b69445700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64060240 con 0x7f7b640f5410 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b62ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b64109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b62ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b64109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49142/0 (socket says 172.21.15.158:49142) 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b62ffd700 1 -- 172.21.15.158:0/3671376234 learned_addr learned my addr 172.21.15.158:0/3671376234 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b62ffd700 1 -- 172.21.15.158:0/3671376234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58005040 con 0x7f7b640f5410 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.946+0000 7f7b62ffd700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b64109520 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7b58005010 tx=0x7f7b5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b58009320 con 0x7f7b640f5410 2024-02-10T09:25:07.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b64060440 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b58013380 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b58014e50 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b64109bd0 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b580095f0 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.947+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b64045ad0 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.948+0000 7f7b4b7fe700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7b4c02ff10 0x7f7b4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.948+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b58040fa0 con 0x7f7b640f5410 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.948+0000 7f7b627fc700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7b4c02ff10 0x7f7b4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.948+0000 7f7b627fc700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7b4c02ff10 0x7f7b4c0323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f7b540096a0 tx=0x7f7b54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:07.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:07.952+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b58014420 con 0x7f7b640f5410 2024-02-10T09:25:08.160 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.159+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b6410a3f0 con 0x7f7b640f5410 2024-02-10T09:25:08.161 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.159+0000 7f7b4b7fe700 1 -- 172.21.15.158:0/3671376234 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b58010230 con 0x7f7b640f5410 2024-02-10T09:25:08.161 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:08.161 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7b4c02ff10 msgr2=0x7f7b4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7b4c02ff10 0x7f7b4c0323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f7b540096a0 tx=0x7f7b54006b40).stop 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 msgr2=0x7f7b64109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b64109520 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7b58005010 tx=0x7f7b5800b040).stop 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 shutdown_connections 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7b4c02ff10 0x7f7b4c0323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 --2- 172.21.15.158:0/3671376234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b640f5410 0x7f7b64109520 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 >> 172.21.15.158:0/3671376234 conn(0x7f7b640f0b70 msgr2=0x7f7b640f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 shutdown_connections 2024-02-10T09:25:08.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:08.161+0000 7f7b69445700 1 -- 172.21.15.158:0/3671376234 wait complete. 2024-02-10T09:25:08.164 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:08.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:06.059937+0000 mgr.smithi083.dowdym (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:09.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:08.160921+0000 mon.smithi083 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.158:0/3671376234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:09.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:09.826 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:10.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:08.060192+0000 mgr.smithi083.dowdym (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:11.962 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 -- 172.21.15.158:0/1131849266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 msgr2=0x7f27540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 --2- 172.21.15.158:0/1131849266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27540f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f2748009fb0 tx=0x7f2748009b30).stop 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 -- 172.21.15.158:0/1131849266 shutdown_connections 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 --2- 172.21.15.158:0/1131849266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27540f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 -- 172.21.15.158:0/1131849266 >> 172.21.15.158:0/1131849266 conn(0x7f27540f0b70 msgr2=0x7f27540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 -- 172.21.15.158:0/1131849266 shutdown_connections 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.960+0000 7f275ab47700 1 -- 172.21.15.158:0/1131849266 wait complete. 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.961+0000 7f275ab47700 1 Processor -- start 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.961+0000 7f275ab47700 1 -- start start 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.961+0000 7f275ab47700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27541056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:11.963 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.961+0000 7f275ab47700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754105bd0 con 0x7f27540f5410 2024-02-10T09:25:11.965 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27588e3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27541056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:11.965 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27588e3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27541056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39746/0 (socket says 172.21.15.158:39746) 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27588e3700 1 -- 172.21.15.158:0/1050705081 learned_addr learned my addr 172.21.15.158:0/1050705081 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27588e3700 1 -- 172.21.15.158:0/1050705081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2748005040 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27588e3700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27541056e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2748005010 tx=0x7f274800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2748009320 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2748013380 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2754105dd0 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2748014e50 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.962+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2754106260 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.963+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27480263a0 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.963+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2754045ad0 con 0x7f27540f5410 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.963+0000 7f27517fa700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f274402ff10 0x7f27440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:11.966 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.963+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27480411c0 con 0x7f27540f5410 2024-02-10T09:25:11.967 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.963+0000 7f2753fff700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f274402ff10 0x7f27440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:11.967 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.964+0000 7f2753fff700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f274402ff10 0x7f27440323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f27400096a0 tx=0x7f2740006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:11.969 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:11.968+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27480126f0 con 0x7f27540f5410 2024-02-10T09:25:12.177 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.175+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2754107110 con 0x7f27540f5410 2024-02-10T09:25:12.178 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.176+0000 7f27517fa700 1 -- 172.21.15.158:0/1050705081 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2748012330 con 0x7f27540f5410 2024-02-10T09:25:12.178 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:12.178 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:12.179 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f274402ff10 msgr2=0x7f27440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:12.180 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f274402ff10 0x7f27440323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f27400096a0 tx=0x7f2740006b40).stop 2024-02-10T09:25:12.180 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 msgr2=0x7f27541056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:12.180 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27541056e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2748005010 tx=0x7f274800b040).stop 2024-02-10T09:25:12.180 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 shutdown_connections 2024-02-10T09:25:12.180 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f274402ff10 0x7f27440323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:12.180 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 --2- 172.21.15.158:0/1050705081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27540f5410 0x7f27541056e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:12.181 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 >> 172.21.15.158:0/1050705081 conn(0x7f27540f0b70 msgr2=0x7f27540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:12.181 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 shutdown_connections 2024-02-10T09:25:12.181 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:12.178+0000 7f275ab47700 1 -- 172.21.15.158:0/1050705081 wait complete. 2024-02-10T09:25:12.181 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:10.060530+0000 mgr.smithi083.dowdym (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:13.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:12 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:12.177541+0000 mon.smithi083 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.158:0/1050705081' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:13.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:13.784 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:14.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:12.060856+0000 mgr.smithi083.dowdym (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:14.168414+0000 mon.smithi083 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:25:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:14.176363+0000 mon.smithi083 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 -- 172.21.15.158:0/2160421204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 msgr2=0x7f5cd00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 --2- 172.21.15.158:0/2160421204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd00f90f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5cc0009fb0 tx=0x7f5cc0009b30).stop 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 -- 172.21.15.158:0/2160421204 shutdown_connections 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 --2- 172.21.15.158:0/2160421204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd00f90f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 -- 172.21.15.158:0/2160421204 >> 172.21.15.158:0/2160421204 conn(0x7f5cd00f0b70 msgr2=0x7f5cd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 -- 172.21.15.158:0/2160421204 shutdown_connections 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 -- 172.21.15.158:0/2160421204 wait complete. 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 Processor -- start 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.429+0000 7f5cd649e700 1 -- start start 2024-02-10T09:25:15.432 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5cd649e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd01067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5cd649e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd0106c90 con 0x7f5cd0060280 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5ccffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd01067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5ccffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd01067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39756/0 (socket says 172.21.15.158:39756) 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5ccffff700 1 -- 172.21.15.158:0/793169632 learned_addr learned my addr 172.21.15.158:0/793169632 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5ccffff700 1 -- 172.21.15.158:0/793169632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc0005040 con 0x7f5cd0060280 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5ccffff700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd01067a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5cc00089f0 tx=0x7f5cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc0009320 con 0x7f5cd0060280 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd0106e90 con 0x7f5cd0060280 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cc0013380 con 0x7f5cd0060280 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.430+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc0014e20 con 0x7f5cd0060280 2024-02-10T09:25:15.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.431+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd0107320 con 0x7f5cd0060280 2024-02-10T09:25:15.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.431+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cc0014420 con 0x7f5cd0060280 2024-02-10T09:25:15.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.431+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd0045ad0 con 0x7f5cd0060280 2024-02-10T09:25:15.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.431+0000 7f5cccff9700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5cb802ff10 0x7f5cb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:15.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.431+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cc0040ff0 con 0x7f5cd0060280 2024-02-10T09:25:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.432+0000 7f5ccf7fe700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5cb802ff10 0x7f5cb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:15.435 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.432+0000 7f5ccf7fe700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5cb802ff10 0x7f5cb80323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5cc40096a0 tx=0x7f5cc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:15.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.436+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cc00095f0 con 0x7f5cd0060280 2024-02-10T09:25:15.641 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.640+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cd01081d0 con 0x7f5cd0060280 2024-02-10T09:25:15.642 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.641+0000 7f5cccff9700 1 -- 172.21.15.158:0/793169632 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cc001db70 con 0x7f5cd0060280 2024-02-10T09:25:15.642 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:15.642 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5cb802ff10 msgr2=0x7f5cb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5cb802ff10 0x7f5cb80323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f5cc40096a0 tx=0x7f5cc4006b40).stop 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 msgr2=0x7f5cd01067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd01067a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5cc00089f0 tx=0x7f5cc000b040).stop 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 shutdown_connections 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5cb802ff10 0x7f5cb80323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 --2- 172.21.15.158:0/793169632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5cd0060280 0x7f5cd01067a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 >> 172.21.15.158:0/793169632 conn(0x7f5cd00f0b70 msgr2=0x7f5cd00f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 shutdown_connections 2024-02-10T09:25:15.644 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:15.642+0000 7f5cd649e700 1 -- 172.21.15.158:0/793169632 wait complete. 2024-02-10T09:25:15.645 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:16.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:14.061162+0000 mgr.smithi083.dowdym (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:16.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:15.641955+0000 mon.smithi083 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.158:0/793169632' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:17.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:17.287 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:16.061538+0000 mgr.smithi083.dowdym (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:19.064 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.058+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/800786441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 msgr2=0x7f6ff00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:19.064 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.058+0000 7f6ff5f8e700 1 --2- 172.21.15.158:0/800786441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff00f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f6fe0009fb0 tx=0x7f6fe0009b30).stop 2024-02-10T09:25:19.064 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/800786441 shutdown_connections 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 --2- 172.21.15.158:0/800786441 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff00f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/800786441 >> 172.21.15.158:0/800786441 conn(0x7f6ff00f1c70 msgr2=0x7f6ff00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/800786441 shutdown_connections 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/800786441 wait complete. 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 Processor -- start 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 -- start start 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.059+0000 7f6ff5f8e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff0106cd0 con 0x7f6ff00f6510 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fef7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fef7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39784/0 (socket says 172.21.15.158:39784) 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fef7fe700 1 -- 172.21.15.158:0/146573132 learned_addr learned my addr 172.21.15.158:0/146573132 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fef7fe700 1 -- 172.21.15.158:0/146573132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe0005040 con 0x7f6ff00f6510 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fef7fe700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff01067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6fe0005010 tx=0x7f6fe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:19.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe0009320 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff0106ed0 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fe0013380 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.060+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fe0014e50 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.061+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff0107360 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.061+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fe0014420 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.061+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff0045ad0 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.061+0000 7f6fd7fff700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6fd802ff10 0x7f6fd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.061+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fe0040f40 con 0x7f6ff00f6510 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.062+0000 7f6feeffd700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6fd802ff10 0x7f6fd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:19.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.062+0000 7f6feeffd700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6fd802ff10 0x7f6fd80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6fe40096a0 tx=0x7f6fe4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:19.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.066+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fe00095f0 con 0x7f6ff00f6510 2024-02-10T09:25:19.273 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.272+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ff0108210 con 0x7f6ff00f6510 2024-02-10T09:25:19.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.272+0000 7f6fd7fff700 1 -- 172.21.15.158:0/146573132 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fe001db70 con 0x7f6ff00f6510 2024-02-10T09:25:19.274 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:19.274 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6fd802ff10 msgr2=0x7f6fd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6fd802ff10 0x7f6fd80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6fe40096a0 tx=0x7f6fe4006b40).stop 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 msgr2=0x7f6ff01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff01067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6fe0005010 tx=0x7f6fe000b040).stop 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 shutdown_connections 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6fd802ff10 0x7f6fd80323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 --2- 172.21.15.158:0/146573132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff00f6510 0x7f6ff01067e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 >> 172.21.15.158:0/146573132 conn(0x7f6ff00f1c70 msgr2=0x7f6ff00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 shutdown_connections 2024-02-10T09:25:19.276 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:19.274+0000 7f6ff5f8e700 1 -- 172.21.15.158:0/146573132 wait complete. 2024-02-10T09:25:19.277 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:18.061845+0000 mgr.smithi083.dowdym (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:19.273923+0000 mon.smithi083 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.158:0/146573132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:20.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:20.814 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:22.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:20.062203+0000 mgr.smithi083.dowdym (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 -- 172.21.15.158:0/3757286014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 msgr2=0x7f10340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 --2- 172.21.15.158:0/3757286014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f10340f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f101c009fb0 tx=0x7f101c009b30).stop 2024-02-10T09:25:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 -- 172.21.15.158:0/3757286014 shutdown_connections 2024-02-10T09:25:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 --2- 172.21.15.158:0/3757286014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f10340f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:22.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 -- 172.21.15.158:0/3757286014 >> 172.21.15.158:0/3757286014 conn(0x7f10340f0b70 msgr2=0x7f10340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 -- 172.21.15.158:0/3757286014 shutdown_connections 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.592+0000 7f1039f9b700 1 -- 172.21.15.158:0/3757286014 wait complete. 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f1039f9b700 1 Processor -- start 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f1039f9b700 1 -- start start 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f1039f9b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f103410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f1039f9b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10341072f0 con 0x7f10340f5410 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f10337fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f103410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f10337fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f103410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:55058/0 (socket says 172.21.15.158:55058) 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.593+0000 7f10337fe700 1 -- 172.21.15.158:0/1976857823 learned_addr learned my addr 172.21.15.158:0/1976857823 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f10337fe700 1 -- 172.21.15.158:0/1976857823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c005040 con 0x7f10340f5410 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f10337fe700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f103410b900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f101c005010 tx=0x7f101c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101c009320 con 0x7f10340f5410 2024-02-10T09:25:22.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1034060220 con 0x7f10340f5410 2024-02-10T09:25:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f101c013380 con 0x7f10340f5410 2024-02-10T09:25:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101c014e50 con 0x7f10340f5410 2024-02-10T09:25:22.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.594+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103410bdf0 con 0x7f10340f5410 2024-02-10T09:25:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.595+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f101c014420 con 0x7f10340f5410 2024-02-10T09:25:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.595+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1034045ad0 con 0x7f10340f5410 2024-02-10T09:25:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.595+0000 7f102bfff700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f102002ff10 0x7f10200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.595+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f101c040e30 con 0x7f10340f5410 2024-02-10T09:25:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.595+0000 7f1032ffd700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f102002ff10 0x7f10200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:22.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.596+0000 7f1032ffd700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f102002ff10 0x7f10200323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f10240096a0 tx=0x7f1024006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:22.601 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.600+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f101c0095f0 con 0x7f10340f5410 2024-02-10T09:25:22.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.803+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f103410c5b0 con 0x7f10340f5410 2024-02-10T09:25:22.805 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.804+0000 7f102bfff700 1 -- 172.21.15.158:0/1976857823 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f101c01db70 con 0x7f10340f5410 2024-02-10T09:25:22.805 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:22.805 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.805+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f102002ff10 msgr2=0x7f10200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.805+0000 7f1039f9b700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f102002ff10 0x7f10200323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f10240096a0 tx=0x7f1024006b40).stop 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.805+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 msgr2=0x7f103410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.805+0000 7f1039f9b700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f103410b900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f101c005010 tx=0x7f101c00b040).stop 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.805+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 shutdown_connections 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.806+0000 7f1039f9b700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f102002ff10 0x7f10200323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.806+0000 7f1039f9b700 1 --2- 172.21.15.158:0/1976857823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10340f5410 0x7f103410b900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.806+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 >> 172.21.15.158:0/1976857823 conn(0x7f10340f0b70 msgr2=0x7f10340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.806+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 shutdown_connections 2024-02-10T09:25:22.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:22.806+0000 7f1039f9b700 1 -- 172.21.15.158:0/1976857823 wait complete. 2024-02-10T09:25:22.808 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:23.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:22.805352+0000 mon.smithi083 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.158:0/1976857823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:22.062570+0000 mgr.smithi083.dowdym (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:24.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:24.437 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:26.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:24.062890+0000 mgr.smithi083.dowdym (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 -- 172.21.15.158:0/3015431320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 msgr2=0x7fee500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 --2- 172.21.15.158:0/3015431320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee500f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fee38009fb0 tx=0x7fee38009b30).stop 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 -- 172.21.15.158:0/3015431320 shutdown_connections 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 --2- 172.21.15.158:0/3015431320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee500f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 -- 172.21.15.158:0/3015431320 >> 172.21.15.158:0/3015431320 conn(0x7fee500f0b70 msgr2=0x7fee500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 -- 172.21.15.158:0/3015431320 shutdown_connections 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.272+0000 7fee54dc0700 1 -- 172.21.15.158:0/3015431320 wait complete. 2024-02-10T09:25:26.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee54dc0700 1 Processor -- start 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee54dc0700 1 -- start start 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee54dc0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee501056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee54dc0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee50105bd0 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee4e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee501056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee4e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee501056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:55086/0 (socket says 172.21.15.158:55086) 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee4e59c700 1 -- 172.21.15.158:0/3900597151 learned_addr learned my addr 172.21.15.158:0/3900597151 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee4e59c700 1 -- 172.21.15.158:0/3900597151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee38005040 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.273+0000 7fee4e59c700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee501056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fee38005010 tx=0x7fee3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.274+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee38009320 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.274+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee50105dd0 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.274+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee38013380 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.274+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee38014e50 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.274+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee50106260 con 0x7fee500f5410 2024-02-10T09:25:26.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.274+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee50045ad0 con 0x7fee500f5410 2024-02-10T09:25:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.275+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee380095f0 con 0x7fee500f5410 2024-02-10T09:25:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.275+0000 7fee46ffd700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee3c02ff10 0x7fee3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.275+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee38040e90 con 0x7fee500f5410 2024-02-10T09:25:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.275+0000 7fee4dd9b700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee3c02ff10 0x7fee3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.275+0000 7fee4dd9b700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee3c02ff10 0x7fee3c0323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fee400096a0 tx=0x7fee40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:26.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.279+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee38014420 con 0x7fee500f5410 2024-02-10T09:25:26.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.487+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee50107110 con 0x7fee500f5410 2024-02-10T09:25:26.489 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.488+0000 7fee46ffd700 1 -- 172.21.15.158:0/3900597151 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee38010230 con 0x7fee500f5410 2024-02-10T09:25:26.489 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:26.489 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.489+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee3c02ff10 msgr2=0x7fee3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.489+0000 7fee54dc0700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee3c02ff10 0x7fee3c0323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fee400096a0 tx=0x7fee40006b40).stop 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.489+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 msgr2=0x7fee501056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.489+0000 7fee54dc0700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee501056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fee38005010 tx=0x7fee3800b040).stop 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.489+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 shutdown_connections 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.489+0000 7fee54dc0700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee3c02ff10 0x7fee3c0323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.490+0000 7fee54dc0700 1 --2- 172.21.15.158:0/3900597151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee500f5410 0x7fee501056e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.490+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 >> 172.21.15.158:0/3900597151 conn(0x7fee500f0b70 msgr2=0x7fee500f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.490+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 shutdown_connections 2024-02-10T09:25:26.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:26.490+0000 7fee54dc0700 1 -- 172.21.15.158:0/3900597151 wait complete. 2024-02-10T09:25:26.492 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:27.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:26.489414+0000 mon.smithi083 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.158:0/3900597151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:28.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:28.164 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:28.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:26.063195+0000 mgr.smithi083.dowdym (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 -- 172.21.15.158:0/4124373223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 msgr2=0x7f93680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 --2- 172.21.15.158:0/4124373223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f935c009fb0 tx=0x7f935c009b30).stop 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 -- 172.21.15.158:0/4124373223 shutdown_connections 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 --2- 172.21.15.158:0/4124373223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 -- 172.21.15.158:0/4124373223 >> 172.21.15.158:0/4124373223 conn(0x7f93680f0b70 msgr2=0x7f93680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 -- 172.21.15.158:0/4124373223 shutdown_connections 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.026+0000 7f936ca29700 1 -- 172.21.15.158:0/4124373223 wait complete. 2024-02-10T09:25:30.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936ca29700 1 Processor -- start 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936ca29700 1 -- start start 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936ca29700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936ca29700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93680fb960 con 0x7f93680f5410 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56488/0 (socket says 172.21.15.158:56488) 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.027+0000 7f936659c700 1 -- 172.21.15.158:0/3615079972 learned_addr learned my addr 172.21.15.158:0/3615079972 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f936659c700 1 -- 172.21.15.158:0/3615079972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c005040 con 0x7f93680f5410 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f936659c700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680fd4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f935c005010 tx=0x7f935c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:30.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c009320 con 0x7f93680f5410 2024-02-10T09:25:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93680fbb00 con 0x7f93680f5410 2024-02-10T09:25:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f935c013380 con 0x7f93680f5410 2024-02-10T09:25:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c014e50 con 0x7f93680f5410 2024-02-10T09:25:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.028+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93680fbf90 con 0x7f93680f5410 2024-02-10T09:25:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.029+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f935c0095f0 con 0x7f93680f5410 2024-02-10T09:25:30.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.029+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9368059100 con 0x7f93680f5410 2024-02-10T09:25:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.029+0000 7f9356ffd700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f935002ff10 0x7f93500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.029+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f935c040fa0 con 0x7f93680f5410 2024-02-10T09:25:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.029+0000 7f9365d9b700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f935002ff10 0x7f93500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:30.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.029+0000 7f9365d9b700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f935002ff10 0x7f93500323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f93580096a0 tx=0x7f9358006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:30.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.033+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f935c014420 con 0x7f93680f5410 2024-02-10T09:25:30.239 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.238+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93680fafa0 con 0x7f93680f5410 2024-02-10T09:25:30.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.239+0000 7f9356ffd700 1 -- 172.21.15.158:0/3615079972 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f935c010230 con 0x7f93680f5410 2024-02-10T09:25:30.240 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:30.240 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.240+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f935002ff10 msgr2=0x7f93500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.240+0000 7f936ca29700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f935002ff10 0x7f93500323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f93580096a0 tx=0x7f9358006b40).stop 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.240+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 msgr2=0x7f93680fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680fd4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f935c005010 tx=0x7f935c00b040).stop 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 shutdown_connections 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f935002ff10 0x7f93500323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:30.242 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 --2- 172.21.15.158:0/3615079972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93680f5410 0x7f93680fd4e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:30.243 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 >> 172.21.15.158:0/3615079972 conn(0x7f93680f0b70 msgr2=0x7f936805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:30.243 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 shutdown_connections 2024-02-10T09:25:30.243 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:30.241+0000 7f936ca29700 1 -- 172.21.15.158:0/3615079972 wait complete. 2024-02-10T09:25:30.243 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:30.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:28.063558+0000 mgr.smithi083.dowdym (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:31.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:30 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:30.240276+0000 mon.smithi083 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.158:0/3615079972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:31.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:31.935 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:32.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:30.063977+0000 mgr.smithi083.dowdym (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.676+0000 7fa8ae232700 1 -- 172.21.15.158:0/2439653822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 msgr2=0x7fa8a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.676+0000 7fa8ae232700 1 --2- 172.21.15.158:0/2439653822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f90f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa898009fb0 tx=0x7fa898009b30).stop 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.676+0000 7fa8ae232700 1 -- 172.21.15.158:0/2439653822 shutdown_connections 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.676+0000 7fa8ae232700 1 --2- 172.21.15.158:0/2439653822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f90f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.676+0000 7fa8ae232700 1 -- 172.21.15.158:0/2439653822 >> 172.21.15.158:0/2439653822 conn(0x7fa8a80f0b70 msgr2=0x7fa8a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.677+0000 7fa8ae232700 1 -- 172.21.15.158:0/2439653822 shutdown_connections 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.677+0000 7fa8ae232700 1 -- 172.21.15.158:0/2439653822 wait complete. 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.677+0000 7fa8ae232700 1 Processor -- start 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.677+0000 7fa8ae232700 1 -- start start 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8ae232700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:33.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8ae232700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a80f4ff0 con 0x7fa8a8060280 2024-02-10T09:25:33.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8a77fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:33.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8a77fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56512/0 (socket says 172.21.15.158:56512) 2024-02-10T09:25:33.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8a77fe700 1 -- 172.21.15.158:0/1991612816 learned_addr learned my addr 172.21.15.158:0/1991612816 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:33.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8a77fe700 1 -- 172.21.15.158:0/1991612816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898005040 con 0x7fa8a8060280 2024-02-10T09:25:33.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8a77fe700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f6b70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa8980089f0 tx=0x7fa89800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:33.681 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa898009320 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a80f5190 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa898013380 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa898014e20 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.678+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a80f5620 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.679+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8980263a0 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.679+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a8059100 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.679+0000 7fa89ffff700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa89402ff10 0x7fa8940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.679+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa898041210 con 0x7fa8a8060280 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.679+0000 7fa8a6ffd700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa89402ff10 0x7fa8940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:33.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.680+0000 7fa8a6ffd700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa89402ff10 0x7fa8940323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa8900096a0 tx=0x7fa890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:33.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.684+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8980126f0 con 0x7fa8a8060280 2024-02-10T09:25:33.890 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.889+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8a80fc090 con 0x7fa8a8060280 2024-02-10T09:25:33.891 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.889+0000 7fa89ffff700 1 -- 172.21.15.158:0/1991612816 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8980126f0 con 0x7fa8a8060280 2024-02-10T09:25:33.891 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:33.891 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:33.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa89402ff10 msgr2=0x7fa8940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa89402ff10 0x7fa8940323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa8900096a0 tx=0x7fa890006b40).stop 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 msgr2=0x7fa8a80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f6b70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa8980089f0 tx=0x7fa89800b040).stop 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 shutdown_connections 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa89402ff10 0x7fa8940323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 --2- 172.21.15.158:0/1991612816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8a8060280 0x7fa8a80f6b70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 >> 172.21.15.158:0/1991612816 conn(0x7fa8a80f0b70 msgr2=0x7fa8a80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 shutdown_connections 2024-02-10T09:25:33.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:33.891+0000 7fa8ae232700 1 -- 172.21.15.158:0/1991612816 wait complete. 2024-02-10T09:25:33.894 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:34.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:32.064359+0000 mgr.smithi083.dowdym (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:35.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:34 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:33.890989+0000 mon.smithi083 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.158:0/1991612816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:35.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:35.565 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:36.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:34.064722+0000 mgr.smithi083.dowdym (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:37.478 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 -- 172.21.15.158:0/3169695772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 msgr2=0x7f72e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:37.478 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 --2- 172.21.15.158:0/3169695772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e80f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f72dc009fb0 tx=0x7f72dc009b30).stop 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 -- 172.21.15.158:0/3169695772 shutdown_connections 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 --2- 172.21.15.158:0/3169695772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e80f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 -- 172.21.15.158:0/3169695772 >> 172.21.15.158:0/3169695772 conn(0x7f72e80f0b70 msgr2=0x7f72e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 -- 172.21.15.158:0/3169695772 shutdown_connections 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.471+0000 7f72ef273700 1 -- 172.21.15.158:0/3169695772 wait complete. 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.472+0000 7f72ef273700 1 Processor -- start 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.472+0000 7f72ef273700 1 -- start start 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.472+0000 7f72ef273700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.472+0000 7f72ef273700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e8105bd0 con 0x7f72e80f5410 2024-02-10T09:25:37.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ed00f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ed00f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56522/0 (socket says 172.21.15.158:56522) 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ed00f700 1 -- 172.21.15.158:0/1708277323 learned_addr learned my addr 172.21.15.158:0/1708277323 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ed00f700 1 -- 172.21.15.158:0/1708277323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc005040 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ed00f700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e81056e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f72dc005010 tx=0x7f72dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72dc009320 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e8105dd0 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72dc013380 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72dc014e50 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.473+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e8106260 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.474+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72dc0095f0 con 0x7f72e80f5410 2024-02-10T09:25:37.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.474+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e8045ad0 con 0x7f72e80f5410 2024-02-10T09:25:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.474+0000 7f72d9ffb700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f72d402ff10 0x7f72d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.475+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72dc040e90 con 0x7f72e80f5410 2024-02-10T09:25:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.475+0000 7f72ec80e700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f72d402ff10 0x7f72d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.475+0000 7f72ec80e700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f72d402ff10 0x7f72d40323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f72e40096a0 tx=0x7f72e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:37.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.479+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72dc014420 con 0x7f72e80f5410 2024-02-10T09:25:37.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.689+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72e8107110 con 0x7f72e80f5410 2024-02-10T09:25:37.691 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.689+0000 7f72d9ffb700 1 -- 172.21.15.158:0/1708277323 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72dc010230 con 0x7f72e80f5410 2024-02-10T09:25:37.691 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:37.691 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:37.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f72d402ff10 msgr2=0x7f72d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:37.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f72d402ff10 0x7f72d40323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f72e40096a0 tx=0x7f72e4006b40).stop 2024-02-10T09:25:37.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 msgr2=0x7f72e81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:37.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e81056e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f72dc005010 tx=0x7f72dc00b040).stop 2024-02-10T09:25:37.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 shutdown_connections 2024-02-10T09:25:37.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f72d402ff10 0x7f72d40323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 --2- 172.21.15.158:0/1708277323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72e80f5410 0x7f72e81056e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 >> 172.21.15.158:0/1708277323 conn(0x7f72e80f0b70 msgr2=0x7f72e80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 shutdown_connections 2024-02-10T09:25:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:37.691+0000 7f72ef273700 1 -- 172.21.15.158:0/1708277323 wait complete. 2024-02-10T09:25:37.693 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:36.064977+0000 mgr.smithi083.dowdym (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:37.690760+0000 mon.smithi083 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.158:0/1708277323' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:39.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:39.363 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:40.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:38.065324+0000 mgr.smithi083.dowdym (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:41.162 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 -- 172.21.15.158:0/3712276103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 msgr2=0x7ff7280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 --2- 172.21.15.158:0/3712276103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff7280f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff71c009fb0 tx=0x7ff71c009b30).stop 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 -- 172.21.15.158:0/3712276103 shutdown_connections 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 --2- 172.21.15.158:0/3712276103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff7280f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 -- 172.21.15.158:0/3712276103 >> 172.21.15.158:0/3712276103 conn(0x7ff7280f0b70 msgr2=0x7ff7280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 -- 172.21.15.158:0/3712276103 shutdown_connections 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.158+0000 7ff72ecbe700 1 -- 172.21.15.158:0/3712276103 wait complete. 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ecbe700 1 Processor -- start 2024-02-10T09:25:41.163 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ecbe700 1 -- start start 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ecbe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff728109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ecbe700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff728060220 con 0x7ff7280f5410 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ca5a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff728109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ca5a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff728109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46078/0 (socket says 172.21.15.158:46078) 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.159+0000 7ff72ca5a700 1 -- 172.21.15.158:0/1229603634 learned_addr learned my addr 172.21.15.158:0/1229603634 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff72ca5a700 1 -- 172.21.15.158:0/1229603634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c005040 con 0x7ff7280f5410 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff72ca5a700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff728109500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff71c005010 tx=0x7ff71c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff71c009320 con 0x7ff7280f5410 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff728060420 con 0x7ff7280f5410 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff71c013380 con 0x7ff7280f5410 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff71c014e50 con 0x7ff7280f5410 2024-02-10T09:25:41.164 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.160+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff728109bb0 con 0x7ff7280f5410 2024-02-10T09:25:41.165 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.161+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff71c014420 con 0x7ff7280f5410 2024-02-10T09:25:41.165 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.161+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff728059100 con 0x7ff7280f5410 2024-02-10T09:25:41.165 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.161+0000 7ff7257fa700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff71802ff10 0x7ff7180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:41.165 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.161+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff71c040f90 con 0x7ff7280f5410 2024-02-10T09:25:41.165 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.161+0000 7ff727fff700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff71802ff10 0x7ff7180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:41.165 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.161+0000 7ff727fff700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff71802ff10 0x7ff7180323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff7140096a0 tx=0x7ff714006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:41.167 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.165+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff71c01d980 con 0x7ff7280f5410 2024-02-10T09:25:41.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.370+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7280fafa0 con 0x7ff7280f5410 2024-02-10T09:25:41.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.371+0000 7ff7257fa700 1 -- 172.21.15.158:0/1229603634 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff71c01d380 con 0x7ff7280f5410 2024-02-10T09:25:41.377 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:41.377 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:41.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff71802ff10 msgr2=0x7ff7180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:41.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff71802ff10 0x7ff7180323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff7140096a0 tx=0x7ff714006b40).stop 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 msgr2=0x7ff728109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff728109500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7ff71c005010 tx=0x7ff71c00b040).stop 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 shutdown_connections 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff71802ff10 0x7ff7180323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 --2- 172.21.15.158:0/1229603634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7280f5410 0x7ff728109500 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 >> 172.21.15.158:0/1229603634 conn(0x7ff7280f0b70 msgr2=0x7ff7280f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 shutdown_connections 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:41.372+0000 7ff72ecbe700 1 -- 172.21.15.158:0/1229603634 wait complete. 2024-02-10T09:25:41.378 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:40.065571+0000 mgr.smithi083.dowdym (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:42.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:41.372027+0000 mon.smithi083 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.158:0/1229603634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:43.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:43.028 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:44.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:42.065867+0000 mgr.smithi083.dowdym (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 -- 172.21.15.158:0/197659476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 msgr2=0x7fe9780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 --2- 172.21.15.158:0/197659476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe9780f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe968009fb0 tx=0x7fe968009b30).stop 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 -- 172.21.15.158:0/197659476 shutdown_connections 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 --2- 172.21.15.158:0/197659476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe9780f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 -- 172.21.15.158:0/197659476 >> 172.21.15.158:0/197659476 conn(0x7fe9780f1c70 msgr2=0x7fe9780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 -- 172.21.15.158:0/197659476 shutdown_connections 2024-02-10T09:25:44.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.803+0000 7fe97fe97700 1 -- 172.21.15.158:0/197659476 wait complete. 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97fe97700 1 Processor -- start 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97fe97700 1 -- start start 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97fe97700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe978106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97fe97700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe978106d00 con 0x7fe9780f6510 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97dc33700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe978106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97dc33700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe978106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46102/0 (socket says 172.21.15.158:46102) 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97dc33700 1 -- 172.21.15.158:0/531359478 learned_addr learned my addr 172.21.15.158:0/531359478 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97dc33700 1 -- 172.21.15.158:0/531359478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968005040 con 0x7fe9780f6510 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.804+0000 7fe97dc33700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe978106810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe968005010 tx=0x7fe96800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe968009320 con 0x7fe9780f6510 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe978106f00 con 0x7fe9780f6510 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe968013380 con 0x7fe9780f6510 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe968014e50 con 0x7fe9780f6510 2024-02-10T09:25:44.807 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe978107390 con 0x7fe9780f6510 2024-02-10T09:25:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9680095f0 con 0x7fe9780f6510 2024-02-10T09:25:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.805+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe978045ad0 con 0x7fe9780f6510 2024-02-10T09:25:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.806+0000 7fe96e7fc700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe96402ff10 0x7fe9640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:44.808 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.806+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe968040fa0 con 0x7fe9780f6510 2024-02-10T09:25:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.806+0000 7fe97d432700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe96402ff10 0x7fe9640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:44.809 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.806+0000 7fe97d432700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe96402ff10 0x7fe9640323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe9740096a0 tx=0x7fe974006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:44.811 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:44.810+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe968014420 con 0x7fe9780f6510 2024-02-10T09:25:45.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.013+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe978108240 con 0x7fe9780f6510 2024-02-10T09:25:45.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.014+0000 7fe96e7fc700 1 -- 172.21.15.158:0/531359478 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe968010230 con 0x7fe9780f6510 2024-02-10T09:25:45.015 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:45.016 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.015+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe96402ff10 msgr2=0x7fe9640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.015+0000 7fe97fe97700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe96402ff10 0x7fe9640323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe9740096a0 tx=0x7fe974006b40).stop 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 msgr2=0x7fe978106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe978106810 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe968005010 tx=0x7fe96800b040).stop 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 shutdown_connections 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe96402ff10 0x7fe9640323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 --2- 172.21.15.158:0/531359478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9780f6510 0x7fe978106810 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 >> 172.21.15.158:0/531359478 conn(0x7fe9780f1c70 msgr2=0x7fe9780f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:45.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 shutdown_connections 2024-02-10T09:25:45.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:45.016+0000 7fe97fe97700 1 -- 172.21.15.158:0/531359478 wait complete. 2024-02-10T09:25:45.018 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:44.066208+0000 mgr.smithi083.dowdym (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:45.015564+0000 mon.smithi083 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.158:0/531359478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:46.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:46.617 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:46.066486+0000 mgr.smithi083.dowdym (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:48.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 -- 172.21.15.158:0/2285058027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 msgr2=0x7f58340fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:48.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 --2- 172.21.15.158:0/2285058027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f58340fb550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f5828009fb0 tx=0x7f5828009b30).stop 2024-02-10T09:25:48.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 -- 172.21.15.158:0/2285058027 shutdown_connections 2024-02-10T09:25:48.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 --2- 172.21.15.158:0/2285058027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f58340fb550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:48.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 -- 172.21.15.158:0/2285058027 >> 172.21.15.158:0/2285058027 conn(0x7f58340f0b70 msgr2=0x7f58340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:48.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 -- 172.21.15.158:0/2285058027 shutdown_connections 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.512+0000 7f583b8c2700 1 -- 172.21.15.158:0/2285058027 wait complete. 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.513+0000 7f583b8c2700 1 Processor -- start 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.513+0000 7f583b8c2700 1 -- start start 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.513+0000 7f583b8c2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f5834108980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.513+0000 7f583b8c2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5834108e70 con 0x7f5834060220 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583965e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f5834108980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583965e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f5834108980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46118/0 (socket says 172.21.15.158:46118) 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583965e700 1 -- 172.21.15.158:0/2988442769 learned_addr learned my addr 172.21.15.158:0/2988442769 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583965e700 1 -- 172.21.15.158:0/2988442769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5828005040 con 0x7f5834060220 2024-02-10T09:25:48.516 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583965e700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f5834108980 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f58280089f0 tx=0x7f582800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5828009320 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5834109070 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5828013380 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5828014e20 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.514+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5834109500 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.515+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f58280095f0 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.515+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834045ad0 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.515+0000 7f5825ffb700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f582002ff60 0x7f5820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.515+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5828040e60 con 0x7f5834060220 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.516+0000 7f5838e5d700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f582002ff60 0x7f5820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:48.518 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.516+0000 7f5838e5d700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f582002ff60 0x7f5820032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f58300096a0 tx=0x7f5830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:48.523 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.520+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5828014420 con 0x7f5834060220 2024-02-10T09:25:48.730 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.729+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f583410a350 con 0x7f5834060220 2024-02-10T09:25:48.731 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.730+0000 7f5825ffb700 1 -- 172.21.15.158:0/2988442769 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5828010230 con 0x7f5834060220 2024-02-10T09:25:48.731 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:48.731 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f582002ff60 msgr2=0x7f5820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f582002ff60 0x7f5820032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f58300096a0 tx=0x7f5830006b40).stop 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 msgr2=0x7f5834108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f5834108980 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f58280089f0 tx=0x7f582800b040).stop 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 shutdown_connections 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f582002ff60 0x7f5820032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 --2- 172.21.15.158:0/2988442769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5834060220 0x7f5834108980 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.731+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 >> 172.21.15.158:0/2988442769 conn(0x7f58340f0b70 msgr2=0x7f58340f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:48.733 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.732+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 shutdown_connections 2024-02-10T09:25:48.734 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:48.732+0000 7f583b8c2700 1 -- 172.21.15.158:0/2988442769 wait complete. 2024-02-10T09:25:48.734 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:49.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:48.731229+0000 mon.smithi083 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.158:0/2988442769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:48.066798+0000 mgr.smithi083.dowdym (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:50.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:50.308 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:52.057 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 -- 172.21.15.158:0/1390134076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 msgr2=0x7f33400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 --2- 172.21.15.158:0/1390134076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33400f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3334009fb0 tx=0x7f3334009b30).stop 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 -- 172.21.15.158:0/1390134076 shutdown_connections 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 --2- 172.21.15.158:0/1390134076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33400f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 -- 172.21.15.158:0/1390134076 >> 172.21.15.158:0/1390134076 conn(0x7f33400f1c70 msgr2=0x7f33400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 -- 172.21.15.158:0/1390134076 shutdown_connections 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.055+0000 7f3346056700 1 -- 172.21.15.158:0/1390134076 wait complete. 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.056+0000 7f3346056700 1 Processor -- start 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.056+0000 7f3346056700 1 -- start start 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.056+0000 7f3346056700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33401067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:52.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.056+0000 7f3346056700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3340106cd0 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f333f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33401067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f333f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33401067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44542/0 (socket says 172.21.15.158:44542) 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f333f7fe700 1 -- 172.21.15.158:0/1266643648 learned_addr learned my addr 172.21.15.158:0/1266643648 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f333f7fe700 1 -- 172.21.15.158:0/1266643648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334005040 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f333f7fe700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33401067e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3334005010 tx=0x7f333400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3334009320 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3340106ed0 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3334013380 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3334014e50 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.057+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3340107360 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.058+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33340263a0 con 0x7f33400f6510 2024-02-10T09:25:52.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.058+0000 7f3327fff700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332802ff10 0x7f33280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.058+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3334041190 con 0x7f33400f6510 2024-02-10T09:25:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.058+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3340045ad0 con 0x7f33400f6510 2024-02-10T09:25:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.059+0000 7f333effd700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332802ff10 0x7f33280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:52.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.059+0000 7f333effd700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332802ff10 0x7f33280323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f33300096a0 tx=0x7f3330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:52.064 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.064+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3334012920 con 0x7f33400f6510 2024-02-10T09:25:52.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.270+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3340108080 con 0x7f33400f6510 2024-02-10T09:25:52.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.271+0000 7f3327fff700 1 -- 172.21.15.158:0/1266643648 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f333402b030 con 0x7f33400f6510 2024-02-10T09:25:52.272 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:52.272 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332802ff10 msgr2=0x7f33280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332802ff10 0x7f33280323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f33300096a0 tx=0x7f3330006b40).stop 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 msgr2=0x7f33401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33401067e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3334005010 tx=0x7f333400b040).stop 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 shutdown_connections 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f332802ff10 0x7f33280323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 --2- 172.21.15.158:0/1266643648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33400f6510 0x7f33401067e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:52.274 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 >> 172.21.15.158:0/1266643648 conn(0x7f33400f1c70 msgr2=0x7f33400f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:52.275 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 shutdown_connections 2024-02-10T09:25:52.275 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:52.272+0000 7f3346056700 1 -- 172.21.15.158:0/1266643648 wait complete. 2024-02-10T09:25:52.275 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:52.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:50.067071+0000 mgr.smithi083.dowdym (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:53.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:52 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:52.272218+0000 mon.smithi083 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.158:0/1266643648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:53.926 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:53.926 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:54.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:52.067332+0000 mgr.smithi083.dowdym (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:55.698 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 -- 172.21.15.158:0/1593611423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 msgr2=0x7f55b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 --2- 172.21.15.158:0/1593611423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b40f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f55a8009fb0 tx=0x7f55a8009b30).stop 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 -- 172.21.15.158:0/1593611423 shutdown_connections 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 --2- 172.21.15.158:0/1593611423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b40f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 -- 172.21.15.158:0/1593611423 >> 172.21.15.158:0/1593611423 conn(0x7f55b40f1c70 msgr2=0x7f55b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 -- 172.21.15.158:0/1593611423 shutdown_connections 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 -- 172.21.15.158:0/1593611423 wait complete. 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 Processor -- start 2024-02-10T09:25:55.699 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.698+0000 7f55baf4a700 1 -- start start 2024-02-10T09:25:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55baf4a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b4108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55baf4a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4108ef0 con 0x7f55b40f6510 2024-02-10T09:25:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55b8ce6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b4108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55b8ce6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b4108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44574/0 (socket says 172.21.15.158:44574) 2024-02-10T09:25:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55b8ce6700 1 -- 172.21.15.158:0/1947190418 learned_addr learned my addr 172.21.15.158:0/1947190418 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:55.701 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55b8ce6700 1 -- 172.21.15.158:0/1947190418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8005040 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.699+0000 7f55b8ce6700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b4108a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f55a8005010 tx=0x7f55a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a8009320 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b41090f0 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55a8013380 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55a8014e50 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b4109580 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55a80095f0 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4045ad0 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.700+0000 7f55b17fa700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f55a402ff10 0x7f55a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.701+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55a8040e90 con 0x7f55b40f6510 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.701+0000 7f55b3fff700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f55a402ff10 0x7f55a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:55.702 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.701+0000 7f55b3fff700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f55a402ff10 0x7f55a40323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f55a00096a0 tx=0x7f55a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:55.705 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.705+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f55a8014420 con 0x7f55b40f6510 2024-02-10T09:25:55.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.908+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55b410a430 con 0x7f55b40f6510 2024-02-10T09:25:55.909 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.908+0000 7f55b17fa700 1 -- 172.21.15.158:0/1947190418 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f55a8010230 con 0x7f55b40f6510 2024-02-10T09:25:55.909 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:55.909 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f55a402ff10 msgr2=0x7f55a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f55a402ff10 0x7f55a40323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f55a00096a0 tx=0x7f55a0006b40).stop 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 msgr2=0x7f55b4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b4108a00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f55a8005010 tx=0x7f55a800b040).stop 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 shutdown_connections 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f55a402ff10 0x7f55a40323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 --2- 172.21.15.158:0/1947190418 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55b40f6510 0x7f55b4108a00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 >> 172.21.15.158:0/1947190418 conn(0x7f55b40f1c70 msgr2=0x7f55b40f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 shutdown_connections 2024-02-10T09:25:55.911 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:55.910+0000 7f55baf4a700 1 -- 172.21.15.158:0/1947190418 wait complete. 2024-02-10T09:25:55.912 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:25:56.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:54.067666+0000 mgr.smithi083.dowdym (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:57.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:56 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:55.908765+0000 mon.smithi083 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.158:0/1947190418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:25:57.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:25:57.477 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:25:58.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:56.067908+0000 mgr.smithi083.dowdym (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:25:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 -- 172.21.15.158:0/178999105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 msgr2=0x7f6c480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 --2- 172.21.15.158:0/178999105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c480f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f6c38009fb0 tx=0x7f6c38009b30).stop 2024-02-10T09:25:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 -- 172.21.15.158:0/178999105 shutdown_connections 2024-02-10T09:25:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 --2- 172.21.15.158:0/178999105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c480f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 -- 172.21.15.158:0/178999105 >> 172.21.15.158:0/178999105 conn(0x7f6c480f0b70 msgr2=0x7f6c480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 -- 172.21.15.158:0/178999105 shutdown_connections 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 -- 172.21.15.158:0/178999105 wait complete. 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 Processor -- start 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.330+0000 7f6c4e197700 1 -- start start 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c4e197700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c48109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c4e197700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c48060220 con 0x7f6c480f5410 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c477fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c48109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c477fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c48109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56492/0 (socket says 172.21.15.158:56492) 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c477fe700 1 -- 172.21.15.158:0/1397028882 learned_addr learned my addr 172.21.15.158:0/1397028882 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c477fe700 1 -- 172.21.15.158:0/1397028882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38005040 con 0x7f6c480f5410 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c477fe700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c48109500 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6c380089f0 tx=0x7f6c3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c38009320 con 0x7f6c480f5410 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c48060420 con 0x7f6c480f5410 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c38013380 con 0x7f6c480f5410 2024-02-10T09:25:59.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.331+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3801ed80 con 0x7f6c480f5410 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.332+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c48109bb0 con 0x7f6c480f5410 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.332+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c3801e420 con 0x7f6c480f5410 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.332+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c48059100 con 0x7f6c480f5410 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.332+0000 7f6c2ffff700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c3002ff60 0x7f6c30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.332+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c38040fc0 con 0x7f6c480f5410 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.333+0000 7f6c46ffd700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c3002ff60 0x7f6c30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.333+0000 7f6c46ffd700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c3002ff60 0x7f6c30032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6c3c0096a0 tx=0x7f6c3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:25:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.337+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c380095f0 con 0x7f6c480f5410 2024-02-10T09:25:59.544 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.544+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c480fafa0 con 0x7f6c480f5410 2024-02-10T09:25:59.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.545+0000 7f6c2ffff700 1 -- 172.21.15.158:0/1397028882 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c3801cb10 con 0x7f6c480f5410 2024-02-10T09:25:59.545 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:25:59.545 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c3002ff60 msgr2=0x7f6c30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c3002ff60 0x7f6c30032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f6c3c0096a0 tx=0x7f6c3c006b40).stop 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 msgr2=0x7f6c48109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c48109500 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f6c380089f0 tx=0x7f6c3800b040).stop 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 shutdown_connections 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c3002ff60 0x7f6c30032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 --2- 172.21.15.158:0/1397028882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c480f5410 0x7f6c48109500 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.546+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 >> 172.21.15.158:0/1397028882 conn(0x7f6c480f0b70 msgr2=0x7f6c480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.547+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 shutdown_connections 2024-02-10T09:25:59.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:25:59.547+0000 7f6c4e197700 1 -- 172.21.15.158:0/1397028882 wait complete. 2024-02-10T09:25:59.548 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:25:58.068241+0000 mgr.smithi083.dowdym (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:25:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:25:59.545151+0000 mon.smithi083 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.158:0/1397028882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:01.247 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:01.247 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:02.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:00.068549+0000 mgr.smithi083.dowdym (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:03.036 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.035+0000 7f5c463f0700 1 -- 172.21.15.158:0/3849085026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 msgr2=0x7f5c400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.035+0000 7f5c463f0700 1 --2- 172.21.15.158:0/3849085026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c400f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5c30009fb0 tx=0x7f5c30009b30).stop 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.035+0000 7f5c463f0700 1 -- 172.21.15.158:0/3849085026 shutdown_connections 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.035+0000 7f5c463f0700 1 --2- 172.21.15.158:0/3849085026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c400f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.035+0000 7f5c463f0700 1 -- 172.21.15.158:0/3849085026 >> 172.21.15.158:0/3849085026 conn(0x7f5c400f1c70 msgr2=0x7f5c400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.035+0000 7f5c463f0700 1 -- 172.21.15.158:0/3849085026 shutdown_connections 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.036+0000 7f5c463f0700 1 -- 172.21.15.158:0/3849085026 wait complete. 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.036+0000 7f5c463f0700 1 Processor -- start 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.036+0000 7f5c463f0700 1 -- start start 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.036+0000 7f5c463f0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c401067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:03.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.036+0000 7f5c463f0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40106cd0 con 0x7f5c400f6510 2024-02-10T09:26:03.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.037+0000 7f5c3ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c401067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:03.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.037+0000 7f5c3ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c401067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56510/0 (socket says 172.21.15.158:56510) 2024-02-10T09:26:03.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.037+0000 7f5c3ffff700 1 -- 172.21.15.158:0/735065925 learned_addr learned my addr 172.21.15.158:0/735065925 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:03.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.037+0000 7f5c3ffff700 1 -- 172.21.15.158:0/735065925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30005040 con 0x7f5c400f6510 2024-02-10T09:26:03.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.037+0000 7f5c3ffff700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c401067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5c30005010 tx=0x7f5c3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:03.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.037+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c30009320 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c30013380 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c30014e50 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c40106ed0 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40107360 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c300095f0 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c40045ad0 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.038+0000 7f5c3cff9700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5c2802ff10 0x7f5c280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.039+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c30040e60 con 0x7f5c400f6510 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.039+0000 7f5c3f7fe700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5c2802ff10 0x7f5c280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:03.040 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.039+0000 7f5c3f7fe700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5c2802ff10 0x7f5c280323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5c340096a0 tx=0x7f5c34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:03.043 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.043+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c3001d980 con 0x7f5c400f6510 2024-02-10T09:26:03.250 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.250+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c40108210 con 0x7f5c400f6510 2024-02-10T09:26:03.251 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.251+0000 7f5c3cff9700 1 -- 172.21.15.158:0/735065925 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5c3001d980 con 0x7f5c400f6510 2024-02-10T09:26:03.251 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:03.251 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5c2802ff10 msgr2=0x7f5c280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5c2802ff10 0x7f5c280323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5c340096a0 tx=0x7f5c34006b40).stop 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 msgr2=0x7f5c401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c401067e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5c30005010 tx=0x7f5c3000b040).stop 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 shutdown_connections 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5c2802ff10 0x7f5c280323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 --2- 172.21.15.158:0/735065925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c400f6510 0x7f5c401067e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 >> 172.21.15.158:0/735065925 conn(0x7f5c400f1c70 msgr2=0x7f5c400f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 shutdown_connections 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:03.252+0000 7f5c463f0700 1 -- 172.21.15.158:0/735065925 wait complete. 2024-02-10T09:26:03.253 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:04.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:02.068809+0000 mgr.smithi083.dowdym (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:04.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:03.251255+0000 mon.smithi083 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.158:0/735065925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:04.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:04.836 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:06.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:04.069160+0000 mgr.smithi083.dowdym (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 -- 172.21.15.158:0/1588029234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 msgr2=0x7faa200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:06.773 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 --2- 172.21.15.158:0/1588029234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa200f55c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7faa1c009fb0 tx=0x7faa1c009b30).stop 2024-02-10T09:26:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 -- 172.21.15.158:0/1588029234 shutdown_connections 2024-02-10T09:26:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 --2- 172.21.15.158:0/1588029234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa200f55c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 -- 172.21.15.158:0/1588029234 >> 172.21.15.158:0/1588029234 conn(0x7faa200f09a0 msgr2=0x7faa200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 -- 172.21.15.158:0/1588029234 shutdown_connections 2024-02-10T09:26:06.774 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 -- 172.21.15.158:0/1588029234 wait complete. 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 Processor -- start 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.773+0000 7faa27081700 1 -- start start 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa27081700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa201054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa27081700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa201059c0 con 0x7faa200f51a0 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa2607f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa201054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa2607f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa201054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56536/0 (socket says 172.21.15.158:56536) 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa2607f700 1 -- 172.21.15.158:0/3747113836 learned_addr learned my addr 172.21.15.158:0/3747113836 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:06.775 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa2607f700 1 -- 172.21.15.158:0/3747113836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c005040 con 0x7faa200f51a0 2024-02-10T09:26:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa2607f700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa201054d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7faa1c0089f0 tx=0x7faa1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.774+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa1c009320 con 0x7faa200f51a0 2024-02-10T09:26:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa20105bc0 con 0x7faa200f51a0 2024-02-10T09:26:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa1c013380 con 0x7faa200f51a0 2024-02-10T09:26:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa1c014e20 con 0x7faa200f51a0 2024-02-10T09:26:06.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa20106050 con 0x7faa200f51a0 2024-02-10T09:26:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa1c0095f0 con 0x7faa200f51a0 2024-02-10T09:26:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa20045ad0 con 0x7faa200f51a0 2024-02-10T09:26:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa16ffd700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faa0c02ff10 0x7faa0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.775+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa1c040f10 con 0x7faa200f51a0 2024-02-10T09:26:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.776+0000 7faa2587e700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faa0c02ff10 0x7faa0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:06.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.776+0000 7faa2587e700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faa0c02ff10 0x7faa0c0323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7faa100096a0 tx=0x7faa10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:06.781 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.780+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa1c014420 con 0x7faa200f51a0 2024-02-10T09:26:06.984 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.984+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa20106f00 con 0x7faa200f51a0 2024-02-10T09:26:06.985 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.985+0000 7faa16ffd700 1 -- 172.21.15.158:0/3747113836 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faa1c01d980 con 0x7faa200f51a0 2024-02-10T09:26:06.985 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:06.986 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faa0c02ff10 msgr2=0x7faa0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faa0c02ff10 0x7faa0c0323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7faa100096a0 tx=0x7faa10006b40).stop 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 msgr2=0x7faa201054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa201054d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7faa1c0089f0 tx=0x7faa1c00b040).stop 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 shutdown_connections 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faa0c02ff10 0x7faa0c0323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 --2- 172.21.15.158:0/3747113836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faa200f51a0 0x7faa201054d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:06.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 >> 172.21.15.158:0/3747113836 conn(0x7faa200f09a0 msgr2=0x7faa200f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:06.988 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.986+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 shutdown_connections 2024-02-10T09:26:06.988 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:06.987+0000 7faa27081700 1 -- 172.21.15.158:0/3747113836 wait complete. 2024-02-10T09:26:06.988 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:08.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:06.069524+0000 mgr.smithi083.dowdym (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:08.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:06.985234+0000 mon.smithi083 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.158:0/3747113836' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:08.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:08.587 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:10.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:08.069870+0000 mgr.smithi083.dowdym (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 -- 172.21.15.158:0/114164251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 msgr2=0x7f45fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 --2- 172.21.15.158:0/114164251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc0f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f45ec009fb0 tx=0x7f45ec009b30).stop 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 -- 172.21.15.158:0/114164251 shutdown_connections 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 --2- 172.21.15.158:0/114164251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc0f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 -- 172.21.15.158:0/114164251 >> 172.21.15.158:0/114164251 conn(0x7f45fc0f0b70 msgr2=0x7f45fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 -- 172.21.15.158:0/114164251 shutdown_connections 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.403+0000 7f460336f700 1 -- 172.21.15.158:0/114164251 wait complete. 2024-02-10T09:26:10.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460336f700 1 Processor -- start 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460336f700 1 -- start start 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460336f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460336f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc106cc0 con 0x7f45fc0f5410 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460110b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460110b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47114/0 (socket says 172.21.15.158:47114) 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460110b700 1 -- 172.21.15.158:0/1797745449 learned_addr learned my addr 172.21.15.158:0/1797745449 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460110b700 1 -- 172.21.15.158:0/1797745449 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec005040 con 0x7f45fc0f5410 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.404+0000 7f460110b700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc1067d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f45ec005010 tx=0x7f45ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.405+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec009320 con 0x7f45fc0f5410 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.405+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc106ec0 con 0x7f45fc0f5410 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.405+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45ec013380 con 0x7f45fc0f5410 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.405+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec014e50 con 0x7f45fc0f5410 2024-02-10T09:26:10.406 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.405+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc107350 con 0x7f45fc0f5410 2024-02-10T09:26:10.407 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.406+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc045ad0 con 0x7f45fc0f5410 2024-02-10T09:26:10.407 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.406+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45ec0095f0 con 0x7f45fc0f5410 2024-02-10T09:26:10.407 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.406+0000 7f45f1ffb700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f45e802ff10 0x7f45e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:10.407 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.406+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45ec040f80 con 0x7f45fc0f5410 2024-02-10T09:26:10.407 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.406+0000 7f460090a700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f45e802ff10 0x7f45e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:10.408 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.406+0000 7f460090a700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f45e802ff10 0x7f45e80323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f45f80096a0 tx=0x7f45f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:10.410 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.410+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45ec014420 con 0x7f45fc0f5410 2024-02-10T09:26:10.614 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.614+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45fc108180 con 0x7f45fc0f5410 2024-02-10T09:26:10.615 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.615+0000 7f45f1ffb700 1 -- 172.21.15.158:0/1797745449 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f45ec010230 con 0x7f45fc0f5410 2024-02-10T09:26:10.615 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:10.615 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:10.616 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f45e802ff10 msgr2=0x7f45e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f45e802ff10 0x7f45e80323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f45f80096a0 tx=0x7f45f8006b40).stop 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 msgr2=0x7f45fc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc1067d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f45ec005010 tx=0x7f45ec00b040).stop 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 shutdown_connections 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f45e802ff10 0x7f45e80323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 --2- 172.21.15.158:0/1797745449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45fc0f5410 0x7f45fc1067d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 >> 172.21.15.158:0/1797745449 conn(0x7f45fc0f0b70 msgr2=0x7f45fc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 shutdown_connections 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:10.616+0000 7f460336f700 1 -- 172.21.15.158:0/1797745449 wait complete. 2024-02-10T09:26:10.617 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:11.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:10.615106+0000 mon.smithi083 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.158:0/1797745449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:10.070199+0000 mgr.smithi083.dowdym (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:12.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:12.335 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:14.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 -- 172.21.15.158:0/518212411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 msgr2=0x7fbe680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:14.233 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 --2- 172.21.15.158:0/518212411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe680f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fbe58009fb0 tx=0x7fbe58009b30).stop 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 -- 172.21.15.158:0/518212411 shutdown_connections 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 --2- 172.21.15.158:0/518212411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe680f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 -- 172.21.15.158:0/518212411 >> 172.21.15.158:0/518212411 conn(0x7fbe680f0b70 msgr2=0x7fbe680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 -- 172.21.15.158:0/518212411 shutdown_connections 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.232+0000 7fbe6e666700 1 -- 172.21.15.158:0/518212411 wait complete. 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.233+0000 7fbe6e666700 1 Processor -- start 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.233+0000 7fbe6e666700 1 -- start start 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.233+0000 7fbe6e666700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe68109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:14.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.233+0000 7fbe6e666700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68060220 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe67fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe68109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe67fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe68109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47136/0 (socket says 172.21.15.158:47136) 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe67fff700 1 -- 172.21.15.158:0/4201016593 learned_addr learned my addr 172.21.15.158:0/4201016593 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe67fff700 1 -- 172.21.15.158:0/4201016593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58005040 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe67fff700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe68109540 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fbe58005010 tx=0x7fbe5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe58009320 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe68060420 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe58013380 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe58014e50 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.234+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe68109bf0 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.235+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe580095f0 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.235+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe68059100 con 0x7fbe680f5410 2024-02-10T09:26:14.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.235+0000 7fbe64ff9700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5002ff10 0x7fbe500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:14.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.235+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe58040e90 con 0x7fbe680f5410 2024-02-10T09:26:14.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.236+0000 7fbe677fe700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5002ff10 0x7fbe500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:14.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.236+0000 7fbe677fe700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5002ff10 0x7fbe500323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fbe5c0096a0 tx=0x7fbe5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:14.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.240+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe58014420 con 0x7fbe680f5410 2024-02-10T09:26:14.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:12.070568+0000 mgr.smithi083.dowdym (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:14.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.445+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe68045ad0 con 0x7fbe680f5410 2024-02-10T09:26:14.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.445+0000 7fbe64ff9700 1 -- 172.21.15.158:0/4201016593 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe58010230 con 0x7fbe680f5410 2024-02-10T09:26:14.446 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:14.446 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5002ff10 msgr2=0x7fbe500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5002ff10 0x7fbe500323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fbe5c0096a0 tx=0x7fbe5c006b40).stop 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 msgr2=0x7fbe68109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe68109540 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fbe58005010 tx=0x7fbe5800b040).stop 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 shutdown_connections 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5002ff10 0x7fbe500323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 --2- 172.21.15.158:0/4201016593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe680f5410 0x7fbe68109540 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 >> 172.21.15.158:0/4201016593 conn(0x7fbe680f0b70 msgr2=0x7fbe680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 shutdown_connections 2024-02-10T09:26:14.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:14.447+0000 7fbe6e666700 1 -- 172.21.15.158:0/4201016593 wait complete. 2024-02-10T09:26:14.449 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:14.169724+0000 mon.smithi083 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:26:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:14.177524+0000 mon.smithi083 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:26:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:14.445776+0000 mon.smithi083 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.158:0/4201016593' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:16.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:16.140 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:16.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:14.070893+0000 mgr.smithi083.dowdym (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 -- 172.21.15.158:0/3536755182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 msgr2=0x7fce3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 --2- 172.21.15.158:0/3536755182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c0f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fce2c009fb0 tx=0x7fce2c009b30).stop 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 -- 172.21.15.158:0/3536755182 shutdown_connections 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 --2- 172.21.15.158:0/3536755182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c0f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 -- 172.21.15.158:0/3536755182 >> 172.21.15.158:0/3536755182 conn(0x7fce3c0f1c70 msgr2=0x7fce3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 -- 172.21.15.158:0/3536755182 shutdown_connections 2024-02-10T09:26:17.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 -- 172.21.15.158:0/3536755182 wait complete. 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 Processor -- start 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.989+0000 7fce41dc7700 1 -- start start 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce41dc7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce41dc7700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c069e20 con 0x7fce3c0f6510 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce3b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce3b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47158/0 (socket says 172.21.15.158:47158) 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce3b7fe700 1 -- 172.21.15.158:0/3239274726 learned_addr learned my addr 172.21.15.158:0/3239274726 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce3b7fe700 1 -- 172.21.15.158:0/3239274726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c005040 con 0x7fce3c0f6510 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce3b7fe700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c06b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fce2c005010 tx=0x7fce2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce2c009320 con 0x7fce3c0f6510 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c069fc0 con 0x7fce3c0f6510 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce2c013380 con 0x7fce3c0f6510 2024-02-10T09:26:17.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce2c014e50 con 0x7fce3c0f6510 2024-02-10T09:26:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.990+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c06a450 con 0x7fce3c0f6510 2024-02-10T09:26:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.991+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce2c0095f0 con 0x7fce3c0f6510 2024-02-10T09:26:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.991+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c059100 con 0x7fce3c0f6510 2024-02-10T09:26:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.991+0000 7fce23fff700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce2402ff10 0x7fce240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.991+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce2c040e70 con 0x7fce3c0f6510 2024-02-10T09:26:17.994 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.992+0000 7fce3affd700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce2402ff10 0x7fce240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:17.995 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.992+0000 7fce3affd700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce2402ff10 0x7fce240323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fce300096a0 tx=0x7fce30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:17.997 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:17.996+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce2c014420 con 0x7fce3c0f6510 2024-02-10T09:26:18.201 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.201+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce3c045ad0 con 0x7fce3c0f6510 2024-02-10T09:26:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.202+0000 7fce23fff700 1 -- 172.21.15.158:0/3239274726 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce2c010230 con 0x7fce3c0f6510 2024-02-10T09:26:18.202 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:18.202 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce2402ff10 msgr2=0x7fce240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce2402ff10 0x7fce240323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fce300096a0 tx=0x7fce30006b40).stop 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 msgr2=0x7fce3c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c06b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fce2c005010 tx=0x7fce2c00b040).stop 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 shutdown_connections 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce2402ff10 0x7fce240323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 --2- 172.21.15.158:0/3239274726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce3c0f6510 0x7fce3c06b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 >> 172.21.15.158:0/3239274726 conn(0x7fce3c0f1c70 msgr2=0x7fce3c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 shutdown_connections 2024-02-10T09:26:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:18.203+0000 7fce41dc7700 1 -- 172.21.15.158:0/3239274726 wait complete. 2024-02-10T09:26:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:18.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:16.071139+0000 mgr.smithi083.dowdym (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:19.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:18 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:18.202034+0000 mon.smithi083 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.158:0/3239274726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:20.052 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:20.052 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:18.071391+0000 mgr.smithi083.dowdym (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 -- 172.21.15.158:0/1201641092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 msgr2=0x7f7d600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 --2- 172.21.15.158:0/1201641092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d600f66c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f7d54009fb0 tx=0x7f7d54009b30).stop 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 -- 172.21.15.158:0/1201641092 shutdown_connections 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 --2- 172.21.15.158:0/1201641092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d600f66c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 -- 172.21.15.158:0/1201641092 >> 172.21.15.158:0/1201641092 conn(0x7f7d600f1aa0 msgr2=0x7f7d600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 -- 172.21.15.158:0/1201641092 shutdown_connections 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.856+0000 7f7d5f59e700 1 -- 172.21.15.158:0/1201641092 wait complete. 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5f59e700 1 Processor -- start 2024-02-10T09:26:21.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5f59e700 1 -- start start 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5f59e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d601065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5f59e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d60106ac0 con 0x7f7d600f62a0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d601065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d601065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57136/0 (socket says 172.21.15.158:57136) 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5e59c700 1 -- 172.21.15.158:0/4090907086 learned_addr learned my addr 172.21.15.158:0/4090907086 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5e59c700 1 -- 172.21.15.158:0/4090907086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54005040 con 0x7f7d600f62a0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.857+0000 7f7d5e59c700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d601065d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f7d540089f0 tx=0x7f7d5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d54009320 con 0x7f7d600f62a0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d54013380 con 0x7f7d600f62a0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d60106cc0 con 0x7f7d600f62a0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d54014e20 con 0x7f7d600f62a0 2024-02-10T09:26:21.859 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d60107150 con 0x7f7d600f62a0 2024-02-10T09:26:21.860 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d60045ad0 con 0x7f7d600f62a0 2024-02-10T09:26:21.860 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.858+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d540095f0 con 0x7f7d600f62a0 2024-02-10T09:26:21.860 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.859+0000 7f7d4effd700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7d4802ff10 0x7f7d480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:21.860 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.859+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d54040e90 con 0x7f7d600f62a0 2024-02-10T09:26:21.860 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.859+0000 7f7d5dd9b700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7d4802ff10 0x7f7d480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:21.861 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.859+0000 7f7d5dd9b700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7d4802ff10 0x7f7d480323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f7d500096a0 tx=0x7f7d50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:21.863 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:21.863+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d54014420 con 0x7f7d600f62a0 2024-02-10T09:26:22.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.068+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d60108000 con 0x7f7d600f62a0 2024-02-10T09:26:22.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.069+0000 7f7d4effd700 1 -- 172.21.15.158:0/4090907086 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7d5401d980 con 0x7f7d600f62a0 2024-02-10T09:26:22.069 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:22.070 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7d4802ff10 msgr2=0x7f7d480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7d4802ff10 0x7f7d480323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f7d500096a0 tx=0x7f7d50006b40).stop 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 msgr2=0x7f7d601065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d601065d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f7d540089f0 tx=0x7f7d5400b040).stop 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 shutdown_connections 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7d4802ff10 0x7f7d480323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 --2- 172.21.15.158:0/4090907086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d600f62a0 0x7f7d601065d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 >> 172.21.15.158:0/4090907086 conn(0x7f7d600f1aa0 msgr2=0x7f7d600f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 shutdown_connections 2024-02-10T09:26:22.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:22.070+0000 7f7d5f59e700 1 -- 172.21.15.158:0/4090907086 wait complete. 2024-02-10T09:26:22.072 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:22.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:20.071750+0000 mgr.smithi083.dowdym (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:23.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:22.069133+0000 mon.smithi083 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.158:0/4090907086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:23.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:23.740 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:22.071993+0000 mgr.smithi083.dowdym (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:25.660 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.657+0000 7fa12017a700 1 -- 172.21.15.158:0/1253712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 msgr2=0x7fa1180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.657+0000 7fa12017a700 1 --2- 172.21.15.158:0/1253712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fa108009fb0 tx=0x7fa108009b30).stop 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.657+0000 7fa12017a700 1 -- 172.21.15.158:0/1253712940 shutdown_connections 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.657+0000 7fa12017a700 1 --2- 172.21.15.158:0/1253712940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.657+0000 7fa12017a700 1 -- 172.21.15.158:0/1253712940 >> 172.21.15.158:0/1253712940 conn(0x7fa1180f0b70 msgr2=0x7fa1180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.658+0000 7fa12017a700 1 -- 172.21.15.158:0/1253712940 shutdown_connections 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.658+0000 7fa12017a700 1 -- 172.21.15.158:0/1253712940 wait complete. 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.658+0000 7fa12017a700 1 Processor -- start 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.658+0000 7fa12017a700 1 -- start start 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.658+0000 7fa12017a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.658+0000 7fa12017a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1180fb960 con 0x7fa1180f5410 2024-02-10T09:26:25.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa11df16700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa11df16700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57160/0 (socket says 172.21.15.158:57160) 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa11df16700 1 -- 172.21.15.158:0/1363013305 learned_addr learned my addr 172.21.15.158:0/1363013305 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa11df16700 1 -- 172.21.15.158:0/1363013305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa108005040 con 0x7fa1180f5410 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa11df16700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180fd4e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa1080089f0 tx=0x7fa10800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa108009320 con 0x7fa1180f5410 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1180fbb00 con 0x7fa1180f5410 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa108013380 con 0x7fa1180f5410 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa10801ed80 con 0x7fa1180f5410 2024-02-10T09:26:25.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.659+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1180fbf90 con 0x7fa1180f5410 2024-02-10T09:26:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.660+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa118059100 con 0x7fa1180f5410 2024-02-10T09:26:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.660+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa10801e420 con 0x7fa1180f5410 2024-02-10T09:26:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.660+0000 7fa10e7fc700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa10402ff60 0x7fa104032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.660+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa108040e70 con 0x7fa1180f5410 2024-02-10T09:26:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.661+0000 7fa11d715700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa10402ff60 0x7fa104032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:25.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.661+0000 7fa11d715700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa10402ff60 0x7fa104032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa1140096a0 tx=0x7fa114006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:25.666 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.665+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa10800e2a0 con 0x7fa1180f5410 2024-02-10T09:26:25.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.868+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1180fafa0 con 0x7fa1180f5410 2024-02-10T09:26:25.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.869+0000 7fa10e7fc700 1 -- 172.21.15.158:0/1363013305 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa10800e2a0 con 0x7fa1180f5410 2024-02-10T09:26:25.870 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:25.870 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa10402ff60 msgr2=0x7fa104032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa10402ff60 0x7fa104032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa1140096a0 tx=0x7fa114006b40).stop 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 msgr2=0x7fa1180fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180fd4e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa1080089f0 tx=0x7fa10800b040).stop 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 shutdown_connections 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa10402ff60 0x7fa104032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 --2- 172.21.15.158:0/1363013305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1180f5410 0x7fa1180fd4e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 >> 172.21.15.158:0/1363013305 conn(0x7fa1180f0b70 msgr2=0x7fa11805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 shutdown_connections 2024-02-10T09:26:25.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:25.870+0000 7fa12017a700 1 -- 172.21.15.158:0/1363013305 wait complete. 2024-02-10T09:26:25.873 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:26.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:24.072295+0000 mgr.smithi083.dowdym (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:26.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:25.870316+0000 mon.smithi083 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.158:0/1363013305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:27.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:27.539 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:28.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:26.072682+0000 mgr.smithi083.dowdym (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.462+0000 7f462d45f700 1 -- 172.21.15.158:0/3197182032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 msgr2=0x7f46280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.462+0000 7f462d45f700 1 --2- 172.21.15.158:0/3197182032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46280f66c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f4618009fb0 tx=0x7f4618009b30).stop 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 -- 172.21.15.158:0/3197182032 shutdown_connections 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 --2- 172.21.15.158:0/3197182032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46280f66c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 -- 172.21.15.158:0/3197182032 >> 172.21.15.158:0/3197182032 conn(0x7f46280f1aa0 msgr2=0x7f46280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 -- 172.21.15.158:0/3197182032 shutdown_connections 2024-02-10T09:26:29.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 -- 172.21.15.158:0/3197182032 wait complete. 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 Processor -- start 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 -- start start 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46281065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.463+0000 7f462d45f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4628106ac0 con 0x7f46280f62a0 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4627fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46281065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4627fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46281065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36104/0 (socket says 172.21.15.158:36104) 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4627fff700 1 -- 172.21.15.158:0/1872389066 learned_addr learned my addr 172.21.15.158:0/1872389066 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4627fff700 1 -- 172.21.15.158:0/1872389066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618005040 con 0x7f46280f62a0 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4627fff700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46281065d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4618009830 tx=0x7f461800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:29.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4618009320 con 0x7f46280f62a0 2024-02-10T09:26:29.468 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4628106cc0 con 0x7f46280f62a0 2024-02-10T09:26:29.468 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4618013380 con 0x7f46280f62a0 2024-02-10T09:26:29.468 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4618014420 con 0x7f46280f62a0 2024-02-10T09:26:29.468 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.464+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4628107150 con 0x7f46280f62a0 2024-02-10T09:26:29.469 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.465+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4618014580 con 0x7f46280f62a0 2024-02-10T09:26:29.469 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.465+0000 7f4624ff9700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f461002ff60 0x7f4610032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:29.469 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.465+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4618040e60 con 0x7f46280f62a0 2024-02-10T09:26:29.469 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.465+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4628045ad0 con 0x7f46280f62a0 2024-02-10T09:26:29.469 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.466+0000 7f46277fe700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f461002ff60 0x7f4610032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:29.469 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.466+0000 7f46277fe700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f461002ff60 0x7f4610032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f461c0096a0 tx=0x7f461c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:29.471 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.470+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46180273f0 con 0x7f46280f62a0 2024-02-10T09:26:29.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.675+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46280fbe30 con 0x7f46280f62a0 2024-02-10T09:26:29.677 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.675+0000 7f4624ff9700 1 -- 172.21.15.158:0/1872389066 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f461801a230 con 0x7f46280f62a0 2024-02-10T09:26:29.677 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:29.677 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:29.678 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f461002ff60 msgr2=0x7f4610032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f461002ff60 0x7f4610032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f461c0096a0 tx=0x7f461c006b40).stop 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 msgr2=0x7f46281065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46281065d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f4618009830 tx=0x7f461800b040).stop 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 shutdown_connections 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f461002ff60 0x7f4610032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 --2- 172.21.15.158:0/1872389066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f46280f62a0 0x7f46281065d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 >> 172.21.15.158:0/1872389066 conn(0x7f46280f1aa0 msgr2=0x7f46280f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 shutdown_connections 2024-02-10T09:26:29.679 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:29.677+0000 7f462d45f700 1 -- 172.21.15.158:0/1872389066 wait complete. 2024-02-10T09:26:29.680 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:30.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:28.073000+0000 mgr.smithi083.dowdym (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:30.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:29.676884+0000 mon.smithi083 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.158:0/1872389066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:31.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:31.339 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:32.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:30.073331+0000 mgr.smithi083.dowdym (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:33.110 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 -- 172.21.15.158:0/2586330279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 msgr2=0x7f93280f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:33.110 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 --2- 172.21.15.158:0/2586330279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93280f7d00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9318009fb0 tx=0x7f9318009b30).stop 2024-02-10T09:26:33.111 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 -- 172.21.15.158:0/2586330279 shutdown_connections 2024-02-10T09:26:33.111 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 --2- 172.21.15.158:0/2586330279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93280f7d00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:33.111 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 -- 172.21.15.158:0/2586330279 >> 172.21.15.158:0/2586330279 conn(0x7f93280f0120 msgr2=0x7f93280f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:33.111 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 -- 172.21.15.158:0/2586330279 shutdown_connections 2024-02-10T09:26:33.111 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 -- 172.21.15.158:0/2586330279 wait complete. 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 Processor -- start 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.109+0000 7f9327fff700 1 -- start start 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9327fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93281065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9327fff700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9328106aa0 con 0x7f93280f78e0 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9326ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93281065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9326ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93281065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36118/0 (socket says 172.21.15.158:36118) 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9326ffd700 1 -- 172.21.15.158:0/3220686920 learned_addr learned my addr 172.21.15.158:0/3220686920 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9326ffd700 1 -- 172.21.15.158:0/3220686920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9318005040 con 0x7f93280f78e0 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9326ffd700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93281065b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f93180089f0 tx=0x7f931800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9318009320 con 0x7f93280f78e0 2024-02-10T09:26:33.113 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9318013380 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9318014e20 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.110+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9328106ca0 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.111+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9328107130 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.111+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93180095f0 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.111+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9328045ad0 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.112+0000 7f930f7fe700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f931002ff60 0x7f9310032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.112+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9318040e40 con 0x7f93280f78e0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.112+0000 7f93267fc700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f931002ff60 0x7f9310032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:33.114 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.112+0000 7f93267fc700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f931002ff60 0x7f9310032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f931c0096a0 tx=0x7f931c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:33.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.116+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9318014420 con 0x7f93280f78e0 2024-02-10T09:26:33.323 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.322+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9328107fe0 con 0x7f93280f78e0 2024-02-10T09:26:33.324 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.323+0000 7f930f7fe700 1 -- 172.21.15.158:0/3220686920 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9318010230 con 0x7f93280f78e0 2024-02-10T09:26:33.324 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:33.324 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f931002ff60 msgr2=0x7f9310032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f931002ff60 0x7f9310032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f931c0096a0 tx=0x7f931c006b40).stop 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 msgr2=0x7f93281065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93281065b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f93180089f0 tx=0x7f931800b040).stop 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 shutdown_connections 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f931002ff60 0x7f9310032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 --2- 172.21.15.158:0/3220686920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93280f78e0 0x7f93281065b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 >> 172.21.15.158:0/3220686920 conn(0x7f93280f0120 msgr2=0x7f93280f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 shutdown_connections 2024-02-10T09:26:33.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:33.324+0000 7f9327fff700 1 -- 172.21.15.158:0/3220686920 wait complete. 2024-02-10T09:26:33.327 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:34.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:32.073697+0000 mgr.smithi083.dowdym (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:34.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:33.324133+0000 mon.smithi083 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.158:0/3220686920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:34.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:34.973 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:36.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:34.074012+0000 mgr.smithi083.dowdym (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:36.891 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2765910299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 msgr2=0x7fdcd00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 --2- 172.21.15.158:0/2765910299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00f55c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fdcc4009fb0 tx=0x7fdcc4009b30).stop 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2765910299 shutdown_connections 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 --2- 172.21.15.158:0/2765910299 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00f55c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2765910299 >> 172.21.15.158:0/2765910299 conn(0x7fdcd00f09a0 msgr2=0x7fdcd00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2765910299 shutdown_connections 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2765910299 wait complete. 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 Processor -- start 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.875+0000 7fdcd5af8700 1 -- start start 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd5af8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd5af8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00fb6f0 con 0x7fdcd00f51a0 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd4af6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:36.892 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd4af6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36136/0 (socket says 172.21.15.158:36136) 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd4af6700 1 -- 172.21.15.158:0/2615381413 learned_addr learned my addr 172.21.15.158:0/2615381413 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd4af6700 1 -- 172.21.15.158:0/2615381413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc4005040 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdcd4af6700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00fd270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fdcc4000c00 tx=0x7fdcc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.876+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc40095f0 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd00fb890 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcc4007e90 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc4014db0 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd00fbd20 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcc4025070 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0059100 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdccd7fa700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdcc002ff60 0x7fdcc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.877+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcc4040970 con 0x7fdcd00f51a0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.878+0000 7fdccffff700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdcc002ff60 0x7fdcc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:36.893 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.878+0000 7fdccffff700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdcc002ff60 0x7fdcc0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fdcbc0096a0 tx=0x7fdcbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:36.894 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:36.882+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcc400f4e0 con 0x7fdcd00f51a0 2024-02-10T09:26:37.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.086+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcd00fad30 con 0x7fdcd00f51a0 2024-02-10T09:26:37.088 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.087+0000 7fdccd7fa700 1 -- 172.21.15.158:0/2615381413 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcc400f300 con 0x7fdcd00f51a0 2024-02-10T09:26:37.088 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:37.088 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdcc002ff60 msgr2=0x7fdcc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdcc002ff60 0x7fdcc0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fdcbc0096a0 tx=0x7fdcbc006b40).stop 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 msgr2=0x7fdcd00fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00fd270 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fdcc4000c00 tx=0x7fdcc400b040).stop 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 shutdown_connections 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdcc002ff60 0x7fdcc0032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:37.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 --2- 172.21.15.158:0/2615381413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdcd00f51a0 0x7fdcd00fd270 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:37.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 >> 172.21.15.158:0/2615381413 conn(0x7fdcd00f09a0 msgr2=0x7fdcd005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:37.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 shutdown_connections 2024-02-10T09:26:37.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:37.089+0000 7fdcd5af8700 1 -- 172.21.15.158:0/2615381413 wait complete. 2024-02-10T09:26:37.091 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:36.074363+0000 mgr.smithi083.dowdym (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:37.088338+0000 mon.smithi083 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.158:0/2615381413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:38.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:38.744 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:40.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:38.074698+0000 mgr.smithi083.dowdym (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:40.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 -- 172.21.15.158:0/2386740520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 msgr2=0x7f77a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 --2- 172.21.15.158:0/2386740520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a80f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f7798009fb0 tx=0x7f7798009b30).stop 2024-02-10T09:26:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 -- 172.21.15.158:0/2386740520 shutdown_connections 2024-02-10T09:26:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 --2- 172.21.15.158:0/2386740520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a80f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 -- 172.21.15.158:0/2386740520 >> 172.21.15.158:0/2386740520 conn(0x7f77a80f0b70 msgr2=0x7f77a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 -- 172.21.15.158:0/2386740520 shutdown_connections 2024-02-10T09:26:40.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.688+0000 7f77b00b6700 1 -- 172.21.15.158:0/2386740520 wait complete. 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77b00b6700 1 Processor -- start 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77b00b6700 1 -- start start 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77b00b6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a8106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77b00b6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a8106d00 con 0x7f77a80f5410 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77ade52700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77ade52700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56414/0 (socket says 172.21.15.158:56414) 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77ade52700 1 -- 172.21.15.158:0/68862580 learned_addr learned my addr 172.21.15.158:0/68862580 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77ade52700 1 -- 172.21.15.158:0/68862580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7798005040 con 0x7f77a80f5410 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f77ade52700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a8106810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7798005010 tx=0x7f779800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.689+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7798009320 con 0x7f77a80f5410 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7798013380 con 0x7f77a80f5410 2024-02-10T09:26:40.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a8106f00 con 0x7f77a80f5410 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7798014e50 con 0x7f77a80f5410 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a8107390 con 0x7f77a80f5410 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7798014420 con 0x7f77a80f5410 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a8045ad0 con 0x7f77a80f5410 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.690+0000 7f779e7fc700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f779402ff10 0x7f77940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.691+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7798040f70 con 0x7f77a80f5410 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.691+0000 7f77ad651700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f779402ff10 0x7f77940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:40.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.691+0000 7f77ad651700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f779402ff10 0x7f77940323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f77a40096a0 tx=0x7f77a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:40.696 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.695+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77980095f0 con 0x7f77a80f5410 2024-02-10T09:26:40.901 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.900+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77a80fafa0 con 0x7f77a80f5410 2024-02-10T09:26:40.902 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.900+0000 7f779e7fc700 1 -- 172.21.15.158:0/68862580 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f779801db70 con 0x7f77a80f5410 2024-02-10T09:26:40.902 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:40.902 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:40.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f779402ff10 msgr2=0x7f77940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f779402ff10 0x7f77940323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f77a40096a0 tx=0x7f77a4006b40).stop 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 msgr2=0x7f77a8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a8106810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f7798005010 tx=0x7f779800b040).stop 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 shutdown_connections 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f779402ff10 0x7f77940323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 --2- 172.21.15.158:0/68862580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77a80f5410 0x7f77a8106810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 >> 172.21.15.158:0/68862580 conn(0x7f77a80f0b70 msgr2=0x7f77a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 shutdown_connections 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:40.902+0000 7f77b00b6700 1 -- 172.21.15.158:0/68862580 wait complete. 2024-02-10T09:26:40.904 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:41.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:40 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:40.901890+0000 mon.smithi083 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.158:0/68862580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:40.075021+0000 mgr.smithi083.dowdym (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:42.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:42.611 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:44.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:42.075355+0000 mgr.smithi083.dowdym (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:44.301 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 -- 172.21.15.158:0/1072450929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 msgr2=0x7fa1fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 --2- 172.21.15.158:0/1072450929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fa1ec009fb0 tx=0x7fa1ec009b30).stop 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 -- 172.21.15.158:0/1072450929 shutdown_connections 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 --2- 172.21.15.158:0/1072450929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 -- 172.21.15.158:0/1072450929 >> 172.21.15.158:0/1072450929 conn(0x7fa1fc0f0b70 msgr2=0x7fa1fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 -- 172.21.15.158:0/1072450929 shutdown_connections 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 -- 172.21.15.158:0/1072450929 wait complete. 2024-02-10T09:26:44.302 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.300+0000 7fa20267e700 1 Processor -- start 2024-02-10T09:26:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa20267e700 1 -- start start 2024-02-10T09:26:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa20267e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa20267e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc106cd0 con 0x7fa1fc0f5410 2024-02-10T09:26:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa1fbfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:44.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa1fbfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56424/0 (socket says 172.21.15.158:56424) 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa1fbfff700 1 -- 172.21.15.158:0/2650418014 learned_addr learned my addr 172.21.15.158:0/2650418014 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa1fbfff700 1 -- 172.21.15.158:0/2650418014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec005040 con 0x7fa1fc0f5410 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.301+0000 7fa1fbfff700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc1067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa1ec005010 tx=0x7fa1ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.302+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1ec009320 con 0x7fa1fc0f5410 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.302+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc106ed0 con 0x7fa1fc0f5410 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.302+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1ec013380 con 0x7fa1fc0f5410 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.302+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1ec014e50 con 0x7fa1fc0f5410 2024-02-10T09:26:44.304 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.302+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc107360 con 0x7fa1fc0f5410 2024-02-10T09:26:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.303+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1ec0095f0 con 0x7fa1fc0f5410 2024-02-10T09:26:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.303+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc045ad0 con 0x7fa1fc0f5410 2024-02-10T09:26:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.303+0000 7fa1f8ff9700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa1e402ff10 0x7fa1e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.303+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1ec040e90 con 0x7fa1fc0f5410 2024-02-10T09:26:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.303+0000 7fa1fb7fe700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa1e402ff10 0x7fa1e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:44.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.303+0000 7fa1fb7fe700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa1e402ff10 0x7fa1e40323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa1f00096a0 tx=0x7fa1f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:44.308 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.307+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1ec014420 con 0x7fa1fc0f5410 2024-02-10T09:26:44.511 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.510+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1fc108210 con 0x7fa1fc0f5410 2024-02-10T09:26:44.512 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.511+0000 7fa1f8ff9700 1 -- 172.21.15.158:0/2650418014 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1ec010230 con 0x7fa1fc0f5410 2024-02-10T09:26:44.512 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:44.512 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa1e402ff10 msgr2=0x7fa1e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa1e402ff10 0x7fa1e40323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa1f00096a0 tx=0x7fa1f0006b40).stop 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 msgr2=0x7fa1fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc1067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fa1ec005010 tx=0x7fa1ec00b040).stop 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 shutdown_connections 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa1e402ff10 0x7fa1e40323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:44.514 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 --2- 172.21.15.158:0/2650418014 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa1fc0f5410 0x7fa1fc1067e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:44.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 >> 172.21.15.158:0/2650418014 conn(0x7fa1fc0f0b70 msgr2=0x7fa1fc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:44.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 shutdown_connections 2024-02-10T09:26:44.515 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:44.513+0000 7fa20267e700 1 -- 172.21.15.158:0/2650418014 wait complete. 2024-02-10T09:26:44.515 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:45.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:44.512402+0000 mon.smithi083 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.158:0/2650418014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:46.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:46.143 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:46.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:44.075648+0000 mgr.smithi083.dowdym (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- 172.21.15.158:0/1681893259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 msgr2=0x7f7e600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 --2- 172.21.15.158:0/1681893259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f7e48009fb0 tx=0x7f7e48009b30).stop 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- 172.21.15.158:0/1681893259 shutdown_connections 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 --2- 172.21.15.158:0/1681893259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- 172.21.15.158:0/1681893259 >> 172.21.15.158:0/1681893259 conn(0x7f7e600f0b70 msgr2=0x7f7e600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- 172.21.15.158:0/1681893259 shutdown_connections 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- 172.21.15.158:0/1681893259 wait complete. 2024-02-10T09:26:47.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 Processor -- start 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- start start 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.973+0000 7f7e64c35700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e600fb960 con 0x7f7e600f5410 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e5e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e5e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56438/0 (socket says 172.21.15.158:56438) 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e5e59c700 1 -- 172.21.15.158:0/1138745380 learned_addr learned my addr 172.21.15.158:0/1138745380 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e5e59c700 1 -- 172.21.15.158:0/1138745380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e48005040 con 0x7f7e600f5410 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e5e59c700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600fd4e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f7e48005010 tx=0x7f7e4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e48009320 con 0x7f7e600f5410 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e600fbb00 con 0x7f7e600f5410 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e48013380 con 0x7f7e600f5410 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e48014e50 con 0x7f7e600f5410 2024-02-10T09:26:47.977 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.974+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e600fbf90 con 0x7f7e600f5410 2024-02-10T09:26:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.975+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e480095f0 con 0x7f7e600f5410 2024-02-10T09:26:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.975+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e60059100 con 0x7f7e600f5410 2024-02-10T09:26:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.975+0000 7f7e56ffd700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7e4c02ff10 0x7f7e4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.975+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e48040e90 con 0x7f7e600f5410 2024-02-10T09:26:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.975+0000 7f7e5dd9b700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7e4c02ff10 0x7f7e4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:47.978 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.976+0000 7f7e5dd9b700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7e4c02ff10 0x7f7e4c0323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7e500096a0 tx=0x7f7e50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:47.981 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:47.980+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e48014420 con 0x7f7e600f5410 2024-02-10T09:26:48.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.185+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e600fafa0 con 0x7f7e600f5410 2024-02-10T09:26:48.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.186+0000 7f7e56ffd700 1 -- 172.21.15.158:0/1138745380 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7e48010230 con 0x7f7e600f5410 2024-02-10T09:26:48.188 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:48.188 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:48.189 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7e4c02ff10 msgr2=0x7f7e4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:48.189 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7e4c02ff10 0x7f7e4c0323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f7e500096a0 tx=0x7f7e50006b40).stop 2024-02-10T09:26:48.189 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 msgr2=0x7f7e600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:48.189 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600fd4e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f7e48005010 tx=0x7f7e4800b040).stop 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 shutdown_connections 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7e4c02ff10 0x7f7e4c0323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 --2- 172.21.15.158:0/1138745380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e600f5410 0x7f7e600fd4e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 >> 172.21.15.158:0/1138745380 conn(0x7f7e600f0b70 msgr2=0x7f7e6005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 shutdown_connections 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:48.188+0000 7f7e64c35700 1 -- 172.21.15.158:0/1138745380 wait complete. 2024-02-10T09:26:48.190 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:46.075953+0000 mgr.smithi083.dowdym (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:49.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:48.187615+0000 mon.smithi083 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.158:0/1138745380' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:49.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:49.865 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:48.076225+0000 mgr.smithi083.dowdym (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 -- 172.21.15.158:0/686628940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 msgr2=0x7fedd40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 --2- 172.21.15.158:0/686628940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd40f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fedc8009fb0 tx=0x7fedc8009b30).stop 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 -- 172.21.15.158:0/686628940 shutdown_connections 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 --2- 172.21.15.158:0/686628940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd40f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 -- 172.21.15.158:0/686628940 >> 172.21.15.158:0/686628940 conn(0x7fedd40f0b70 msgr2=0x7fedd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 -- 172.21.15.158:0/686628940 shutdown_connections 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.579+0000 7feddc39d700 1 -- 172.21.15.158:0/686628940 wait complete. 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.580+0000 7feddc39d700 1 Processor -- start 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.580+0000 7feddc39d700 1 -- start start 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.580+0000 7feddc39d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:51.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.580+0000 7feddc39d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4105bd0 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedda139700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedda139700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44626/0 (socket says 172.21.15.158:44626) 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedda139700 1 -- 172.21.15.158:0/4135889220 learned_addr learned my addr 172.21.15.158:0/4135889220 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedda139700 1 -- 172.21.15.158:0/4135889220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8005040 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedda139700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd41056e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fedc8005010 tx=0x7fedc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc8009320 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd4105dd0 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedc8013380 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.581+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc8014e50 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.582+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd4106260 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.582+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fedc80095f0 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.582+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd4045ad0 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.583+0000 7fedc6ffd700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fedc002ff10 0x7fedc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.583+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fedc8040e90 con 0x7fedd40f5410 2024-02-10T09:26:51.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.583+0000 7fedd9938700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fedc002ff10 0x7fedc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:51.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.583+0000 7fedd9938700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fedc002ff10 0x7fedc00323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fedd00096a0 tx=0x7fedd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:51.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.587+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fedc8014420 con 0x7fedd40f5410 2024-02-10T09:26:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.792+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fedd4107110 con 0x7fedd40f5410 2024-02-10T09:26:51.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.793+0000 7fedc6ffd700 1 -- 172.21.15.158:0/4135889220 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fedc8010230 con 0x7fedd40f5410 2024-02-10T09:26:51.795 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:51.795 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:51.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fedc002ff10 msgr2=0x7fedc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:51.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fedc002ff10 0x7fedc00323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fedd00096a0 tx=0x7fedd0006b40).stop 2024-02-10T09:26:51.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 msgr2=0x7fedd41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:51.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd41056e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fedc8005010 tx=0x7fedc800b040).stop 2024-02-10T09:26:51.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 shutdown_connections 2024-02-10T09:26:51.797 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fedc002ff10 0x7fedc00323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:51.797 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 --2- 172.21.15.158:0/4135889220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fedd40f5410 0x7fedd41056e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:51.797 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 >> 172.21.15.158:0/4135889220 conn(0x7fedd40f0b70 msgr2=0x7fedd40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:51.797 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 shutdown_connections 2024-02-10T09:26:51.797 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:51.795+0000 7feddc39d700 1 -- 172.21.15.158:0/4135889220 wait complete. 2024-02-10T09:26:51.797 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:52.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:50.076500+0000 mgr.smithi083.dowdym (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:52.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:51.794548+0000 mon.smithi083 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.158:0/4135889220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:53.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:53.449 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:54.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:52.076838+0000 mgr.smithi083.dowdym (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:55.368 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 -- 172.21.15.158:0/2384474480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 msgr2=0x7f559c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 --2- 172.21.15.158:0/2384474480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c0f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f5590009fb0 tx=0x7f5590009b30).stop 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 -- 172.21.15.158:0/2384474480 shutdown_connections 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 --2- 172.21.15.158:0/2384474480 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c0f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 -- 172.21.15.158:0/2384474480 >> 172.21.15.158:0/2384474480 conn(0x7f559c0f0b70 msgr2=0x7f559c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 -- 172.21.15.158:0/2384474480 shutdown_connections 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 -- 172.21.15.158:0/2384474480 wait complete. 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 Processor -- start 2024-02-10T09:26:55.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.367+0000 7f55a2b05700 1 -- start start 2024-02-10T09:26:55.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a2b05700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:55.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a2b05700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559c060240 con 0x7f559c0f5410 2024-02-10T09:26:55.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a08a1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:55.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a08a1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44646/0 (socket says 172.21.15.158:44646) 2024-02-10T09:26:55.371 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a08a1700 1 -- 172.21.15.158:0/2843019839 learned_addr learned my addr 172.21.15.158:0/2843019839 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a08a1700 1 -- 172.21.15.158:0/2843019839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5590005040 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a08a1700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c1094e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5590005010 tx=0x7f559000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5590009320 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c060440 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5590013380 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5590014e50 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.368+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559c109b90 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.369+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55900263a0 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.369+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559c045ad0 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.369+0000 7f55997fa700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f558c02ff10 0x7f558c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.369+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5590041250 con 0x7f559c0f5410 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.370+0000 7f559bfff700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f558c02ff10 0x7f558c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:55.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.370+0000 7f559bfff700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f558c02ff10 0x7f558c0323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f55880096a0 tx=0x7f5588006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:55.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.374+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f559001cb10 con 0x7f559c0f5410 2024-02-10T09:26:55.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.577+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f559c10a3b0 con 0x7f559c0f5410 2024-02-10T09:26:55.579 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.577+0000 7f55997fa700 1 -- 172.21.15.158:0/2843019839 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f559001cb10 con 0x7f559c0f5410 2024-02-10T09:26:55.579 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:55.579 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:55.580 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f558c02ff10 msgr2=0x7f558c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:55.580 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f558c02ff10 0x7f558c0323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f55880096a0 tx=0x7f5588006b40).stop 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 msgr2=0x7f559c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c1094e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f5590005010 tx=0x7f559000b040).stop 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 shutdown_connections 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f558c02ff10 0x7f558c0323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 --2- 172.21.15.158:0/2843019839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f559c0f5410 0x7f559c1094e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 >> 172.21.15.158:0/2843019839 conn(0x7f559c0f0b70 msgr2=0x7f559c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 shutdown_connections 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:55.579+0000 7f55a2b05700 1 -- 172.21.15.158:0/2843019839 wait complete. 2024-02-10T09:26:55.581 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:26:56.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:54.077168+0000 mgr.smithi083.dowdym (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:56.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:55.578833+0000 mon.smithi083 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.158:0/2843019839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:26:57.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:26:57.205 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:26:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:56.077504+0000 mgr.smithi083.dowdym (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.987+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3747163343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 msgr2=0x7f8c340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.987+0000 7f8c39ecf700 1 --2- 172.21.15.158:0/3747163343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c340f66c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f8c28009fb0 tx=0x7f8c28009b30).stop 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.987+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3747163343 shutdown_connections 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.987+0000 7f8c39ecf700 1 --2- 172.21.15.158:0/3747163343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c340f66c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3747163343 >> 172.21.15.158:0/3747163343 conn(0x7f8c340f1aa0 msgr2=0x7f8c340f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3747163343 shutdown_connections 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3747163343 wait complete. 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 Processor -- start 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 -- start start 2024-02-10T09:26:58.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c341087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.988+0000 7f8c39ecf700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c34108cd0 con 0x7f8c340f62a0 2024-02-10T09:26:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c38ecd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c341087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c38ecd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c341087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44654/0 (socket says 172.21.15.158:44654) 2024-02-10T09:26:58.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c38ecd700 1 -- 172.21.15.158:0/3899176518 learned_addr learned my addr 172.21.15.158:0/3899176518 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:26:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c38ecd700 1 -- 172.21.15.158:0/3899176518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c28005040 con 0x7f8c340f62a0 2024-02-10T09:26:58.992 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c38ecd700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c341087e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f8c280089f0 tx=0x7f8c2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c28009320 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c34108ed0 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c28013380 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c28014e20 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.989+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c34109360 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.990+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c280095f0 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.990+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c34045ad0 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.990+0000 7f8c317fa700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8c2402ff10 0x7f8c240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.990+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c28040e90 con 0x7f8c340f62a0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.991+0000 7f8c33fff700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8c2402ff10 0x7f8c240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:26:58.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.991+0000 7f8c33fff700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8c2402ff10 0x7f8c240323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f8c200096a0 tx=0x7f8c20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:26:58.996 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:58.995+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c28014420 con 0x7f8c340f62a0 2024-02-10T09:26:59.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.199+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c3410a210 con 0x7f8c340f62a0 2024-02-10T09:26:59.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.200+0000 7f8c317fa700 1 -- 172.21.15.158:0/3899176518 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c2801d980 con 0x7f8c340f62a0 2024-02-10T09:26:59.205 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:26:59.205 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:26:59.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8c2402ff10 msgr2=0x7f8c240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:59.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8c2402ff10 0x7f8c240323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f8c200096a0 tx=0x7f8c20006b40).stop 2024-02-10T09:26:59.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 msgr2=0x7f8c341087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c341087e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f8c280089f0 tx=0x7f8c2800b040).stop 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 shutdown_connections 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8c2402ff10 0x7f8c240323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 --2- 172.21.15.158:0/3899176518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c340f62a0 0x7f8c341087e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 >> 172.21.15.158:0/3899176518 conn(0x7f8c340f1aa0 msgr2=0x7f8c340f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.201+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 shutdown_connections 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:26:59.202+0000 7f8c39ecf700 1 -- 172.21.15.158:0/3899176518 wait complete. 2024-02-10T09:26:59.206 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:26:58.077756+0000 mgr.smithi083.dowdym (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:26:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:26:59.201295+0000 mon.smithi083 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.158:0/3899176518' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:00.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:00.860 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:02.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:00.078024+0000 mgr.smithi083.dowdym (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 -- 172.21.15.158:0/2701705955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 msgr2=0x7fcf700f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 --2- 172.21.15.158:0/2701705955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f91b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fcf60009fb0 tx=0x7fcf60009b30).stop 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 -- 172.21.15.158:0/2701705955 shutdown_connections 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 --2- 172.21.15.158:0/2701705955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f91b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 -- 172.21.15.158:0/2701705955 >> 172.21.15.158:0/2701705955 conn(0x7fcf700f0b70 msgr2=0x7fcf700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 -- 172.21.15.158:0/2701705955 shutdown_connections 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 -- 172.21.15.158:0/2701705955 wait complete. 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 Processor -- start 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.684+0000 7fcf7771f700 1 -- start start 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf7771f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:02.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf7771f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700f50b0 con 0x7fcf700601c0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf754bb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf754bb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:32884/0 (socket says 172.21.15.158:32884) 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf754bb700 1 -- 172.21.15.158:0/2048372580 learned_addr learned my addr 172.21.15.158:0/2048372580 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf754bb700 1 -- 172.21.15.158:0/2048372580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf60005040 con 0x7fcf700601c0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf754bb700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f6c30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fcf60009830 tx=0x7fcf6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf60009320 con 0x7fcf700601c0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf700f5250 con 0x7fcf700601c0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf60013380 con 0x7fcf700601c0 2024-02-10T09:27:02.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.685+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf60014420 con 0x7fcf700601c0 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.686+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf700f56e0 con 0x7fcf700601c0 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.686+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf6001e420 con 0x7fcf700601c0 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.686+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf70059100 con 0x7fcf700601c0 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.687+0000 7fcf65ffb700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcf5c02ff60 0x7fcf5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.687+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf60040ee0 con 0x7fcf700601c0 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.687+0000 7fcf74cba700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcf5c02ff60 0x7fcf5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:02.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.687+0000 7fcf74cba700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcf5c02ff60 0x7fcf5c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fcf6c0096a0 tx=0x7fcf6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:02.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.691+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf6001c930 con 0x7fcf700601c0 2024-02-10T09:27:02.900 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.899+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf700fc1e0 con 0x7fcf700601c0 2024-02-10T09:27:02.901 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.900+0000 7fcf65ffb700 1 -- 172.21.15.158:0/2048372580 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf6002c2d0 con 0x7fcf700601c0 2024-02-10T09:27:02.901 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:02.901 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcf5c02ff60 msgr2=0x7fcf5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcf5c02ff60 0x7fcf5c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fcf6c0096a0 tx=0x7fcf6c006b40).stop 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 msgr2=0x7fcf700f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f6c30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fcf60009830 tx=0x7fcf6000b040).stop 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 shutdown_connections 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcf5c02ff60 0x7fcf5c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 --2- 172.21.15.158:0/2048372580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf700601c0 0x7fcf700f6c30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 >> 172.21.15.158:0/2048372580 conn(0x7fcf700f0b70 msgr2=0x7fcf700f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 shutdown_connections 2024-02-10T09:27:02.903 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:02.901+0000 7fcf7771f700 1 -- 172.21.15.158:0/2048372580 wait complete. 2024-02-10T09:27:02.904 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:03.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:02 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:02.901069+0000 mon.smithi083 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.158:0/2048372580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:04.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:02.078205+0000 mgr.smithi083.dowdym (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:04.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:04.551 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:06.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:04.078547+0000 mgr.smithi083.dowdym (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:06.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 -- 172.21.15.158:0/3795915652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 msgr2=0x7f11c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 --2- 172.21.15.158:0/3795915652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c40f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f11b4009fb0 tx=0x7f11b4009b30).stop 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 -- 172.21.15.158:0/3795915652 shutdown_connections 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 --2- 172.21.15.158:0/3795915652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c40f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 -- 172.21.15.158:0/3795915652 >> 172.21.15.158:0/3795915652 conn(0x7f11c40f0b70 msgr2=0x7f11c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 -- 172.21.15.158:0/3795915652 shutdown_connections 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.447+0000 7f11ca8c0700 1 -- 172.21.15.158:0/3795915652 wait complete. 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11ca8c0700 1 Processor -- start 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11ca8c0700 1 -- start start 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11ca8c0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11ca8c0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c4106cd0 con 0x7f11c40f5410 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11c3fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11c3fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:32894/0 (socket says 172.21.15.158:32894) 2024-02-10T09:27:06.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11c3fff700 1 -- 172.21.15.158:0/2121313682 learned_addr learned my addr 172.21.15.158:0/2121313682 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.448+0000 7f11c3fff700 1 -- 172.21.15.158:0/2121313682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b4005040 con 0x7f11c40f5410 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.449+0000 7f11c3fff700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c41067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f11b4005010 tx=0x7f11b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.449+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11b4009320 con 0x7f11c40f5410 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.449+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11b4013380 con 0x7f11c40f5410 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.449+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c4106ed0 con 0x7f11c40f5410 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.449+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11b4014e50 con 0x7f11c40f5410 2024-02-10T09:27:06.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.449+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c4107360 con 0x7f11c40f5410 2024-02-10T09:27:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.450+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11b4014420 con 0x7f11c40f5410 2024-02-10T09:27:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.450+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4045ad0 con 0x7f11c40f5410 2024-02-10T09:27:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.450+0000 7f11c0ff9700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f11ac02ff10 0x7f11ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.450+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11b4040f70 con 0x7f11c40f5410 2024-02-10T09:27:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.450+0000 7f11c37fe700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f11ac02ff10 0x7f11ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:06.453 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.450+0000 7f11c37fe700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f11ac02ff10 0x7f11ac0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f11b80096a0 tx=0x7f11b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:06.456 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.454+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11b40095f0 con 0x7f11c40f5410 2024-02-10T09:27:06.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.660+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11c40fafa0 con 0x7f11c40f5410 2024-02-10T09:27:06.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.661+0000 7f11c0ff9700 1 -- 172.21.15.158:0/2121313682 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11b401db70 con 0x7f11c40f5410 2024-02-10T09:27:06.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:06.663 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:06.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f11ac02ff10 msgr2=0x7f11ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:06.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f11ac02ff10 0x7f11ac0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f11b80096a0 tx=0x7f11b8006b40).stop 2024-02-10T09:27:06.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 msgr2=0x7f11c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:06.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c41067e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f11b4005010 tx=0x7f11b400b040).stop 2024-02-10T09:27:06.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 shutdown_connections 2024-02-10T09:27:06.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f11ac02ff10 0x7f11ac0323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:06.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 --2- 172.21.15.158:0/2121313682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c40f5410 0x7f11c41067e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:06.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 >> 172.21.15.158:0/2121313682 conn(0x7f11c40f0b70 msgr2=0x7f11c405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:06.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 shutdown_connections 2024-02-10T09:27:06.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:06.663+0000 7f11ca8c0700 1 -- 172.21.15.158:0/2121313682 wait complete. 2024-02-10T09:27:06.665 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:07.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:06 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:06.662420+0000 mon.smithi083 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.158:0/2121313682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:08.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:06.078944+0000 mgr.smithi083.dowdym (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:08.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:08.322 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:10.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:08.079313+0000 mgr.smithi083.dowdym (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 -- 172.21.15.158:0/1930854576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 msgr2=0x7f08d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 --2- 172.21.15.158:0/1930854576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d00f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f08c4009fb0 tx=0x7f08c4009b30).stop 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 -- 172.21.15.158:0/1930854576 shutdown_connections 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 --2- 172.21.15.158:0/1930854576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d00f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 -- 172.21.15.158:0/1930854576 >> 172.21.15.158:0/1930854576 conn(0x7f08d00f0b70 msgr2=0x7f08d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 -- 172.21.15.158:0/1930854576 shutdown_connections 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.290+0000 7f08d4bb9700 1 -- 172.21.15.158:0/1930854576 wait complete. 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.291+0000 7f08d4bb9700 1 Processor -- start 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.291+0000 7f08d4bb9700 1 -- start start 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.291+0000 7f08d4bb9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:10.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.291+0000 7f08d4bb9700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0105bd0 con 0x7f08d00f5410 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08ce59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08ce59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:38894/0 (socket says 172.21.15.158:38894) 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08ce59c700 1 -- 172.21.15.158:0/2597996461 learned_addr learned my addr 172.21.15.158:0/2597996461 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08ce59c700 1 -- 172.21.15.158:0/2597996461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c4005040 con 0x7f08d00f5410 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08ce59c700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d01056e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f08c4005010 tx=0x7f08c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08c4009320 con 0x7f08d00f5410 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08c4013380 con 0x7f08d00f5410 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d0105dd0 con 0x7f08d00f5410 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08c4014e50 con 0x7f08d00f5410 2024-02-10T09:27:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.292+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d0106260 con 0x7f08d00f5410 2024-02-10T09:27:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.293+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08c4014420 con 0x7f08d00f5410 2024-02-10T09:27:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.293+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d0045ad0 con 0x7f08d00f5410 2024-02-10T09:27:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.293+0000 7f08beffd700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f08b802ff10 0x7f08b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.293+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08c4040e30 con 0x7f08d00f5410 2024-02-10T09:27:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.294+0000 7f08cdd9b700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f08b802ff10 0x7f08b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.294+0000 7f08cdd9b700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f08b802ff10 0x7f08b80323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08c00096a0 tx=0x7f08c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:10.299 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.298+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08c40095f0 con 0x7f08d00f5410 2024-02-10T09:27:10.501 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.500+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08d0107110 con 0x7f08d00f5410 2024-02-10T09:27:10.502 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.500+0000 7f08beffd700 1 -- 172.21.15.158:0/2597996461 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f08c401db70 con 0x7f08d00f5410 2024-02-10T09:27:10.502 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:10.502 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:10.503 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f08b802ff10 msgr2=0x7f08b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f08b802ff10 0x7f08b80323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f08c00096a0 tx=0x7f08c0006b40).stop 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 msgr2=0x7f08d01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d01056e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f08c4005010 tx=0x7f08c400b040).stop 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 shutdown_connections 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f08b802ff10 0x7f08b80323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 --2- 172.21.15.158:0/2597996461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08d00f5410 0x7f08d01056e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 >> 172.21.15.158:0/2597996461 conn(0x7f08d00f0b70 msgr2=0x7f08d00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 shutdown_connections 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:10.502+0000 7f08d4bb9700 1 -- 172.21.15.158:0/2597996461 wait complete. 2024-02-10T09:27:10.504 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:11.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:10.501963+0000 mon.smithi083 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.158:0/2597996461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:12.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:12.142 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:10.079641+0000 mgr.smithi083.dowdym (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 -- 172.21.15.158:0/3832065220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 msgr2=0x7fea980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 --2- 172.21.15.158:0/3832065220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea980f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7fea8c009fb0 tx=0x7fea8c009b30).stop 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 -- 172.21.15.158:0/3832065220 shutdown_connections 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 --2- 172.21.15.158:0/3832065220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea980f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 -- 172.21.15.158:0/3832065220 >> 172.21.15.158:0/3832065220 conn(0x7fea980f0b70 msgr2=0x7fea980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 -- 172.21.15.158:0/3832065220 shutdown_connections 2024-02-10T09:27:13.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 -- 172.21.15.158:0/3832065220 wait complete. 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.936+0000 7fea9dbe3700 1 Processor -- start 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea9dbe3700 1 -- start start 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea9dbe3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea981056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea9dbe3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea98105bd0 con 0x7fea980f5410 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea977fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea981056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea977fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea981056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:38918/0 (socket says 172.21.15.158:38918) 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea977fe700 1 -- 172.21.15.158:0/4168788956 learned_addr learned my addr 172.21.15.158:0/4168788956 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea977fe700 1 -- 172.21.15.158:0/4168788956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c005040 con 0x7fea980f5410 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.937+0000 7fea977fe700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea981056e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fea8c005010 tx=0x7fea8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea8c009320 con 0x7fea980f5410 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea8c013380 con 0x7fea980f5410 2024-02-10T09:27:13.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea98105dd0 con 0x7fea980f5410 2024-02-10T09:27:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea8c014e50 con 0x7fea980f5410 2024-02-10T09:27:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea98106260 con 0x7fea980f5410 2024-02-10T09:27:13.941 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea8c0095f0 con 0x7fea980f5410 2024-02-10T09:27:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.938+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea98045ad0 con 0x7fea980f5410 2024-02-10T09:27:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.939+0000 7fea7ffff700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fea8002ff10 0x7fea800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.939+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea8c040f10 con 0x7fea980f5410 2024-02-10T09:27:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.939+0000 7fea96ffd700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fea8002ff10 0x7fea800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:13.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.939+0000 7fea96ffd700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fea8002ff10 0x7fea800323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fea880096a0 tx=0x7fea88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:13.944 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:13.943+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea8c014420 con 0x7fea980f5410 2024-02-10T09:27:14.150 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.148+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea98107110 con 0x7fea980f5410 2024-02-10T09:27:14.151 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.149+0000 7fea7ffff700 1 -- 172.21.15.158:0/4168788956 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea8c010230 con 0x7fea980f5410 2024-02-10T09:27:14.151 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:14.151 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:14.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fea8002ff10 msgr2=0x7fea800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:14.152 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fea8002ff10 0x7fea800323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fea880096a0 tx=0x7fea88006b40).stop 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 msgr2=0x7fea981056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea981056e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fea8c005010 tx=0x7fea8c00b040).stop 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 shutdown_connections 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fea8002ff10 0x7fea800323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 --2- 172.21.15.158:0/4168788956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fea980f5410 0x7fea981056e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 >> 172.21.15.158:0/4168788956 conn(0x7fea980f0b70 msgr2=0x7fea980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 shutdown_connections 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:14.151+0000 7fea9dbe3700 1 -- 172.21.15.158:0/4168788956 wait complete. 2024-02-10T09:27:14.153 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:14.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:12.080028+0000 mgr.smithi083.dowdym (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:15.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:14.150619+0000 mon.smithi083 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.158:0/4168788956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:14.170538+0000 mon.smithi083 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:27:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:14.178574+0000 mon.smithi083 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:27:15.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:15.830 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:14.080316+0000 mgr.smithi083.dowdym (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.565+0000 7f3b18872700 1 -- 172.21.15.158:0/452141069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 msgr2=0x7f3b140f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.565+0000 7f3b18872700 1 --2- 172.21.15.158:0/452141069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f4b60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f3b08009fb0 tx=0x7f3b08009b30).stop 2024-02-10T09:27:17.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 -- 172.21.15.158:0/452141069 shutdown_connections 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 --2- 172.21.15.158:0/452141069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f4b60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 -- 172.21.15.158:0/452141069 >> 172.21.15.158:0/452141069 conn(0x7f3b140f0260 msgr2=0x7f3b140f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 -- 172.21.15.158:0/452141069 shutdown_connections 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 -- 172.21.15.158:0/452141069 wait complete. 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 Processor -- start 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 -- start start 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b18872700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140f5770 con 0x7f3b140f4740 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b1259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.566+0000 7f3b1259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:38950/0 (socket says 172.21.15.158:38950) 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b1259c700 1 -- 172.21.15.158:0/4131850047 learned_addr learned my addr 172.21.15.158:0/4131850047 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b1259c700 1 -- 172.21.15.158:0/4131850047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b08005040 con 0x7f3b140f4740 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b1259c700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f72f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3b08005010 tx=0x7f3b0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:17.574 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b08009320 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b140f5910 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b08013380 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b08014e50 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.567+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b140f5da0 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.568+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b08014420 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.568+0000 7f3b02ffd700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3afc02ff10 0x7f3afc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.568+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b14059100 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.568+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b08040e30 con 0x7f3b140f4740 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.568+0000 7f3b11d9b700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3afc02ff10 0x7f3afc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.569+0000 7f3b11d9b700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3afc02ff10 0x7f3afc0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3b040096a0 tx=0x7f3b04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:17.575 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.573+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b080095f0 con 0x7f3b140f4740 2024-02-10T09:27:17.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.778+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b140fc080 con 0x7f3b140f4740 2024-02-10T09:27:17.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.778+0000 7f3b02ffd700 1 -- 172.21.15.158:0/4131850047 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b08007d90 con 0x7f3b140f4740 2024-02-10T09:27:17.780 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:17.780 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3afc02ff10 msgr2=0x7f3afc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3afc02ff10 0x7f3afc0323d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3b040096a0 tx=0x7f3b04006b40).stop 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 msgr2=0x7f3b140f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f72f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3b08005010 tx=0x7f3b0800b040).stop 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 shutdown_connections 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3afc02ff10 0x7f3afc0323d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 --2- 172.21.15.158:0/4131850047 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3b140f4740 0x7f3b140f72f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 >> 172.21.15.158:0/4131850047 conn(0x7f3b140f0260 msgr2=0x7f3b140f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 shutdown_connections 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:17.780+0000 7f3b18872700 1 -- 172.21.15.158:0/4131850047 wait complete. 2024-02-10T09:27:17.782 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:16.080697+0000 mgr.smithi083.dowdym (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:18.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:17.779815+0000 mon.smithi083 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.158:0/4131850047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:19.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:19.422 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:18.081120+0000 mgr.smithi083.dowdym (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:21.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 -- 172.21.15.158:0/3024597136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 msgr2=0x7ff3a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 --2- 172.21.15.158:0/3024597136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a00f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7ff390009fb0 tx=0x7ff390009b30).stop 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 -- 172.21.15.158:0/3024597136 shutdown_connections 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 --2- 172.21.15.158:0/3024597136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a00f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 -- 172.21.15.158:0/3024597136 >> 172.21.15.158:0/3024597136 conn(0x7ff3a00f0b70 msgr2=0x7ff3a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 -- 172.21.15.158:0/3024597136 shutdown_connections 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.315+0000 7ff3a81fc700 1 -- 172.21.15.158:0/3024597136 wait complete. 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a81fc700 1 Processor -- start 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a81fc700 1 -- start start 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a81fc700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a0109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a81fc700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0060240 con 0x7ff3a00f5410 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a5f98700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a0109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a5f98700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a0109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:41666/0 (socket says 172.21.15.158:41666) 2024-02-10T09:27:21.319 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.316+0000 7ff3a5f98700 1 -- 172.21.15.158:0/2412429237 learned_addr learned my addr 172.21.15.158:0/2412429237 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3a5f98700 1 -- 172.21.15.158:0/2412429237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390005040 con 0x7ff3a00f5410 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3a5f98700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a0109520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7ff390005010 tx=0x7ff39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff390009320 con 0x7ff3a00f5410 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a0060440 con 0x7ff3a00f5410 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff390013380 con 0x7ff3a00f5410 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff390014e50 con 0x7ff3a00f5410 2024-02-10T09:27:21.320 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.317+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a0109bd0 con 0x7ff3a00f5410 2024-02-10T09:27:21.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.318+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff390014420 con 0x7ff3a00f5410 2024-02-10T09:27:21.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.318+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a0045ad0 con 0x7ff3a00f5410 2024-02-10T09:27:21.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.318+0000 7ff3967fc700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff38c02ff10 0x7ff38c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:21.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.318+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff390040eb0 con 0x7ff3a00f5410 2024-02-10T09:27:21.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.318+0000 7ff3a5797700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff38c02ff10 0x7ff38c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:21.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.319+0000 7ff3a5797700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff38c02ff10 0x7ff38c0323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff39c0096a0 tx=0x7ff39c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:21.324 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.323+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3900095f0 con 0x7ff3a00f5410 2024-02-10T09:27:21.530 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.529+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3a010a3f0 con 0x7ff3a00f5410 2024-02-10T09:27:21.531 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3967fc700 1 -- 172.21.15.158:0/2412429237 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff39001db70 con 0x7ff3a00f5410 2024-02-10T09:27:21.531 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:21.531 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff38c02ff10 msgr2=0x7ff38c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff38c02ff10 0x7ff38c0323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7ff39c0096a0 tx=0x7ff39c006b40).stop 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 msgr2=0x7ff3a0109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a0109520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7ff390005010 tx=0x7ff39000b040).stop 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 shutdown_connections 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff38c02ff10 0x7ff38c0323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 --2- 172.21.15.158:0/2412429237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3a00f5410 0x7ff3a0109520 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.531+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 >> 172.21.15.158:0/2412429237 conn(0x7ff3a00f0b70 msgr2=0x7ff3a00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:21.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.532+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 shutdown_connections 2024-02-10T09:27:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:21.532+0000 7ff3a81fc700 1 -- 172.21.15.158:0/2412429237 wait complete. 2024-02-10T09:27:21.534 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:22.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:20.081542+0000 mgr.smithi083.dowdym (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:22.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:21.531266+0000 mon.smithi083 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.158:0/2412429237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:23.203 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:23.204 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:24.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:22.081845+0000 mgr.smithi083.dowdym (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:25.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 -- 172.21.15.158:0/2958787486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 msgr2=0x7fa4640f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 --2- 172.21.15.158:0/2958787486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4640f9190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fa458009fb0 tx=0x7fa458009b30).stop 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 -- 172.21.15.158:0/2958787486 shutdown_connections 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 --2- 172.21.15.158:0/2958787486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4640f9190 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 -- 172.21.15.158:0/2958787486 >> 172.21.15.158:0/2958787486 conn(0x7fa4640f0b70 msgr2=0x7fa4640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 -- 172.21.15.158:0/2958787486 shutdown_connections 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 -- 172.21.15.158:0/2958787486 wait complete. 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.064+0000 7fa469cf3700 1 Processor -- start 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa469cf3700 1 -- start start 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa469cf3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4641067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa469cf3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa464106cc0 con 0x7fa4640601e0 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa4637fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4641067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa4637fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4641067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:41682/0 (socket says 172.21.15.158:41682) 2024-02-10T09:27:25.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa4637fe700 1 -- 172.21.15.158:0/3604003572 learned_addr learned my addr 172.21.15.158:0/3604003572 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa4637fe700 1 -- 172.21.15.158:0/3604003572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa458005040 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.065+0000 7fa4637fe700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4641067d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fa458009830 tx=0x7fa458009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.066+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa458018070 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.066+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa458013410 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.066+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa464106ec0 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.066+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa45801c3c0 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.066+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa464107350 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.066+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa45801d420 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.067+0000 7fa44bfff700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa44c02ff60 0x7fa44c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.067+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa458041550 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.067+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa464045ad0 con 0x7fa4640601e0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.067+0000 7fa462ffd700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa44c02ff60 0x7fa44c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.067+0000 7fa462ffd700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa44c02ff60 0x7fa44c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa4540096a0 tx=0x7fa454006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:25.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.071+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa458013ca0 con 0x7fa4640601e0 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:27:23.985397+0000 mgr.smithi083.dowdym (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi083 ... 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:23:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:23:40Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:24:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:25:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:26:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:26:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: main() 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: r = ctx.func(ctx) 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: return func(ctx) 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: raise RuntimeError('uid/gid not found') 2024-02-10T09:27:25.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: RuntimeError: uid/gid not found 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: yield (conn, connr) 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: code, '\n'.join(err))) 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi083 ... 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:23:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:23:40Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:24:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:25:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:26:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:26:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: main() 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: r = ctx.func(ctx) 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: return func(ctx) 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-10T09:27:25.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:27:25.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: raise RuntimeError('uid/gid not found') 2024-02-10T09:27:25.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: RuntimeError: uid/gid not found 2024-02-10T09:27:25.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:23.988278+0000 mon.smithi083 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T09:27:25.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:23.990937+0000 mon.smithi083 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi083", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T09:27:25.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:23.991668+0000 mon.smithi083 (mon.0) 203 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:27:25.281 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:27:23.992246+0000 mgr.smithi083.dowdym (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi083 on smithi083 2024-02-10T09:27:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.278+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4640fc0a0 con 0x7fa4640601e0 2024-02-10T09:27:25.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.278+0000 7fa44bfff700 1 -- 172.21.15.158:0/3604003572 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa45801b370 con 0x7fa4640601e0 2024-02-10T09:27:25.282 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:25.282 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa44c02ff60 msgr2=0x7fa44c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa44c02ff60 0x7fa44c032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa4540096a0 tx=0x7fa454006b40).stop 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 msgr2=0x7fa4641067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4641067d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fa458009830 tx=0x7fa458009b30).stop 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 shutdown_connections 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa44c02ff60 0x7fa44c032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 --2- 172.21.15.158:0/3604003572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa4640601e0 0x7fa4641067d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 >> 172.21.15.158:0/3604003572 conn(0x7fa4640f0b70 msgr2=0x7fa4640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 shutdown_connections 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:25.280+0000 7fa469cf3700 1 -- 172.21.15.158:0/3604003572 wait complete. 2024-02-10T09:27:25.283 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:26.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:24.082111+0000 mgr.smithi083.dowdym (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:26.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:25.279021+0000 mon.smithi083 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.158:0/3604003572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:26.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:26.855 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:26.082343+0000 mgr.smithi083.dowdym (mgr.14162) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:26.273469+0000 mon.smithi083 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:26.327949+0000 mon.smithi083 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:26.332951+0000 mon.smithi083 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:26.333396+0000 mon.smithi083 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:26.333817+0000 mgr.smithi083.dowdym (mgr.14162) 160 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:26.339035+0000 mon.smithi083 (mon.0) 209 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:27:27.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:27:26.341923+0000 mgr.smithi083.dowdym (mgr.14162) 161 : cephadm [INF] Deploying daemon grafana.smithi083 on smithi083 2024-02-10T09:27:28.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 -- 172.21.15.158:0/1374209888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 msgr2=0x7fbe700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:28.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 --2- 172.21.15.158:0/1374209888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe700f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fbe58009fb0 tx=0x7fbe58009b30).stop 2024-02-10T09:27:28.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 -- 172.21.15.158:0/1374209888 shutdown_connections 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 --2- 172.21.15.158:0/1374209888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe700f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 -- 172.21.15.158:0/1374209888 >> 172.21.15.158:0/1374209888 conn(0x7fbe700f1c70 msgr2=0x7fbe700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 -- 172.21.15.158:0/1374209888 shutdown_connections 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.684+0000 7fbe751c0700 1 -- 172.21.15.158:0/1374209888 wait complete. 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.685+0000 7fbe751c0700 1 Processor -- start 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.685+0000 7fbe751c0700 1 -- start start 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.685+0000 7fbe751c0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:28.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.685+0000 7fbe751c0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70106cd0 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe6ed9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe6ed9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:41690/0 (socket says 172.21.15.158:41690) 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe6ed9d700 1 -- 172.21.15.158:0/1877291520 learned_addr learned my addr 172.21.15.158:0/1877291520 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe6ed9d700 1 -- 172.21.15.158:0/1877291520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58005040 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe6ed9d700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe701067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fbe580089f0 tx=0x7fbe5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe58009320 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70106ed0 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe58013380 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe5801ed80 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.686+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe70107360 con 0x7fbe700f6510 2024-02-10T09:27:28.688 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.687+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe5801e420 con 0x7fbe700f6510 2024-02-10T09:27:28.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.687+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe70045ad0 con 0x7fbe700f6510 2024-02-10T09:27:28.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.687+0000 7fbe677fe700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5c02ff60 0x7fbe5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:28.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.687+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe58040ed0 con 0x7fbe700f6510 2024-02-10T09:27:28.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.687+0000 7fbe6e59c700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5c02ff60 0x7fbe5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:28.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.688+0000 7fbe6e59c700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5c02ff60 0x7fbe5c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fbe600096a0 tx=0x7fbe60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:28.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.692+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe5801c930 con 0x7fbe700f6510 2024-02-10T09:27:28.896 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.896+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe70108210 con 0x7fbe700f6510 2024-02-10T09:27:28.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.897+0000 7fbe677fe700 1 -- 172.21.15.158:0/1877291520 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe5801cb40 con 0x7fbe700f6510 2024-02-10T09:27:28.897 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:28.897 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5c02ff60 msgr2=0x7fbe5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:28.898 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5c02ff60 0x7fbe5c032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fbe600096a0 tx=0x7fbe60006b40).stop 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 msgr2=0x7fbe701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe701067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fbe580089f0 tx=0x7fbe5800b040).stop 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 shutdown_connections 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fbe5c02ff60 0x7fbe5c032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 --2- 172.21.15.158:0/1877291520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbe700f6510 0x7fbe701067e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 >> 172.21.15.158:0/1877291520 conn(0x7fbe700f1c70 msgr2=0x7fbe700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 shutdown_connections 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:28.898+0000 7fbe751c0700 1 -- 172.21.15.158:0/1877291520 wait complete. 2024-02-10T09:27:28.899 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:29.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:28.082736+0000 mgr.smithi083.dowdym (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:29.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:28.896902+0000 mon.smithi083 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.158:0/1877291520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:30.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:30.489 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:32.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:30.083029+0000 mgr.smithi083.dowdym (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:32.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 -- 172.21.15.158:0/3776535832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 msgr2=0x7f75840f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:32.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 --2- 172.21.15.158:0/3776535832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f8fc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7580009fb0 tx=0x7f7580009b30).stop 2024-02-10T09:27:32.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 -- 172.21.15.158:0/3776535832 shutdown_connections 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 --2- 172.21.15.158:0/3776535832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f8fc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 -- 172.21.15.158:0/3776535832 >> 172.21.15.158:0/3776535832 conn(0x7f75840f09a0 msgr2=0x7f75840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 -- 172.21.15.158:0/3776535832 shutdown_connections 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.267+0000 7f758a06f700 1 -- 172.21.15.158:0/3776535832 wait complete. 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758a06f700 1 Processor -- start 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758a06f700 1 -- start start 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758a06f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758a06f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75840f4ec0 con 0x7f7584060140 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758906d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758906d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47964/0 (socket says 172.21.15.158:47964) 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758906d700 1 -- 172.21.15.158:0/3309215111 learned_addr learned my addr 172.21.15.158:0/3309215111 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.268+0000 7f758906d700 1 -- 172.21.15.158:0/3309215111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7580005040 con 0x7f7584060140 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.269+0000 7f758906d700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f6a40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f75800089f0 tx=0x7f758000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:32.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.269+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7580009320 con 0x7f7584060140 2024-02-10T09:27:32.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.269+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7580013380 con 0x7f7584060140 2024-02-10T09:27:32.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.269+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75840f5060 con 0x7f7584060140 2024-02-10T09:27:32.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.269+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f758001ed80 con 0x7f7584060140 2024-02-10T09:27:32.271 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.269+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75840f54f0 con 0x7f7584060140 2024-02-10T09:27:32.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.270+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7584059100 con 0x7f7584060140 2024-02-10T09:27:32.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.270+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f758001e420 con 0x7f7584060140 2024-02-10T09:27:32.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.270+0000 7f7579ffb700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f757002ffb0 0x7f7570032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:32.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.270+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7580040f00 con 0x7f7584060140 2024-02-10T09:27:32.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.270+0000 7f758886c700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f757002ffb0 0x7f7570032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:32.272 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.271+0000 7f758886c700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f757002ffb0 0x7f7570032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f75740096a0 tx=0x7f7574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:32.275 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.275+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75800095f0 con 0x7f7584060140 2024-02-10T09:27:32.477 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.477+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7584045ad0 con 0x7f7584060140 2024-02-10T09:27:32.478 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.478+0000 7f7579ffb700 1 -- 172.21.15.158:0/3309215111 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f758001cb10 con 0x7f7584060140 2024-02-10T09:27:32.478 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:32.479 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f757002ffb0 msgr2=0x7f7570032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f757002ffb0 0x7f7570032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f75740096a0 tx=0x7f7574006b40).stop 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 msgr2=0x7f75840f6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f6a40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f75800089f0 tx=0x7f758000b040).stop 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 shutdown_connections 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f757002ffb0 0x7f7570032470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:32.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 --2- 172.21.15.158:0/3309215111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7584060140 0x7f75840f6a40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:32.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 >> 172.21.15.158:0/3309215111 conn(0x7f75840f09a0 msgr2=0x7f75840f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:32.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 shutdown_connections 2024-02-10T09:27:32.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:32.480+0000 7f758a06f700 1 -- 172.21.15.158:0/3309215111 wait complete. 2024-02-10T09:27:32.481 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:33.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:32 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:32.478359+0000 mon.smithi083 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.158:0/3309215111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:34.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:32.083330+0000 mgr.smithi083.dowdym (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:34.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:34.048 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 -- 172.21.15.158:0/3365133975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 msgr2=0x7fee200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 --2- 172.21.15.158:0/3365133975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee200f66c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fee10009fb0 tx=0x7fee10009b30).stop 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 -- 172.21.15.158:0/3365133975 shutdown_connections 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 --2- 172.21.15.158:0/3365133975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee200f66c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 -- 172.21.15.158:0/3365133975 >> 172.21.15.158:0/3365133975 conn(0x7fee200f1aa0 msgr2=0x7fee200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 -- 172.21.15.158:0/3365133975 shutdown_connections 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.861+0000 7fee2667a700 1 -- 172.21.15.158:0/3365133975 wait complete. 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.862+0000 7fee2667a700 1 Processor -- start 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.862+0000 7fee2667a700 1 -- start start 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.862+0000 7fee2667a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee201065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.862+0000 7fee2667a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20106ac0 con 0x7fee200f62a0 2024-02-10T09:27:35.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.862+0000 7fee25678700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee201065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee25678700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee201065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47976/0 (socket says 172.21.15.158:47976) 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee25678700 1 -- 172.21.15.158:0/4076175747 learned_addr learned my addr 172.21.15.158:0/4076175747 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee25678700 1 -- 172.21.15.158:0/4076175747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10005040 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee25678700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee201065d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fee100089f0 tx=0x7fee1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee10009320 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee10013380 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee10014e20 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20106cc0 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.863+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20107150 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.864+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee100095f0 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.864+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee20045ad0 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.864+0000 7fee15ffb700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee0c02ff10 0x7fee0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.864+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee10040e90 con 0x7fee200f62a0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.864+0000 7fee24e77700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee0c02ff10 0x7fee0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:35.883 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.865+0000 7fee24e77700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee0c02ff10 0x7fee0c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fee1c0096a0 tx=0x7fee1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:35.884 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:35.869+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee10014420 con 0x7fee200f62a0 2024-02-10T09:27:36.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:34.083562+0000 mgr.smithi083.dowdym (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:36.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.076+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee20108000 con 0x7fee200f62a0 2024-02-10T09:27:36.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.076+0000 7fee15ffb700 1 -- 172.21.15.158:0/4076175747 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee1001d980 con 0x7fee200f62a0 2024-02-10T09:27:36.076 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:36.077 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:36.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee0c02ff10 msgr2=0x7fee0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:36.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee0c02ff10 0x7fee0c0323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fee1c0096a0 tx=0x7fee1c006b40).stop 2024-02-10T09:27:36.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 msgr2=0x7fee201065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee201065d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fee100089f0 tx=0x7fee1000b040).stop 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 shutdown_connections 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fee0c02ff10 0x7fee0c0323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 --2- 172.21.15.158:0/4076175747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee200f62a0 0x7fee201065d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 >> 172.21.15.158:0/4076175747 conn(0x7fee200f1aa0 msgr2=0x7fee200f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 shutdown_connections 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:36.078+0000 7fee2667a700 1 -- 172.21.15.158:0/4076175747 wait complete. 2024-02-10T09:27:36.079 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:37.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:36.076747+0000 mon.smithi083 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.158:0/4076175747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:37.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:37.695 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:38.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:36.083786+0000 mgr.smithi083.dowdym (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 -- 172.21.15.158:0/2601662929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 msgr2=0x7f9f900f4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 --2- 172.21.15.158:0/2601662929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f900f4b00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f9f8c009fb0 tx=0x7f9f8c009b30).stop 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 -- 172.21.15.158:0/2601662929 shutdown_connections 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 --2- 172.21.15.158:0/2601662929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f900f4b00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 -- 172.21.15.158:0/2601662929 >> 172.21.15.158:0/2601662929 conn(0x7f9f900f0120 msgr2=0x7f9f900f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 -- 172.21.15.158:0/2601662929 shutdown_connections 2024-02-10T09:27:39.576 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.574+0000 7f9f971de700 1 -- 172.21.15.158:0/2601662929 wait complete. 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f971de700 1 Processor -- start 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f971de700 1 -- start start 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f971de700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f90108800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f971de700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f90108cf0 con 0x7f9f900f46c0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f961dc700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f90108800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f961dc700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f90108800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56998/0 (socket says 172.21.15.158:56998) 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f961dc700 1 -- 172.21.15.158:0/282274589 learned_addr learned my addr 172.21.15.158:0/282274589 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.575+0000 7f9f961dc700 1 -- 172.21.15.158:0/282274589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c005040 con 0x7f9f900f46c0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f961dc700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f90108800 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f9f8c005010 tx=0x7f9f8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f8c009320 con 0x7f9f900f46c0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f8c013380 con 0x7f9f900f46c0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f90108ef0 con 0x7f9f900f46c0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f8c014e50 con 0x7f9f900f46c0 2024-02-10T09:27:39.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f90109380 con 0x7f9f900f46c0 2024-02-10T09:27:39.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.576+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9f8c014420 con 0x7f9f900f46c0 2024-02-10T09:27:39.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.577+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f90045ad0 con 0x7f9f900f46c0 2024-02-10T09:27:39.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.577+0000 7f9f86ffd700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9f7c02ff10 0x7f9f7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:39.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.577+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9f8c040f40 con 0x7f9f900f46c0 2024-02-10T09:27:39.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.577+0000 7f9f959db700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9f7c02ff10 0x7f9f7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:39.578 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.577+0000 7f9f959db700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9f7c02ff10 0x7f9f7c0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9f800096a0 tx=0x7f9f80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:39.581 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.581+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9f8c0095f0 con 0x7f9f900f46c0 2024-02-10T09:27:39.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.785+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f9010a230 con 0x7f9f900f46c0 2024-02-10T09:27:39.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.786+0000 7f9f86ffd700 1 -- 172.21.15.158:0/282274589 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9f8c01db70 con 0x7f9f900f46c0 2024-02-10T09:27:39.786 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:39.786 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:39.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9f7c02ff10 msgr2=0x7f9f7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:39.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9f7c02ff10 0x7f9f7c0323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9f800096a0 tx=0x7f9f80006b40).stop 2024-02-10T09:27:39.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 msgr2=0x7f9f90108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:39.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f90108800 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f9f8c005010 tx=0x7f9f8c00b040).stop 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 shutdown_connections 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9f7c02ff10 0x7f9f7c0323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 --2- 172.21.15.158:0/282274589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f900f46c0 0x7f9f90108800 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.787+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 >> 172.21.15.158:0/282274589 conn(0x7f9f900f0120 msgr2=0x7f9f900f0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.788+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 shutdown_connections 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:39.788+0000 7f9f971de700 1 -- 172.21.15.158:0/282274589 wait complete. 2024-02-10T09:27:39.789 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:40.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:38.083997+0000 mgr.smithi083.dowdym (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:41.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:40 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:39.786373+0000 mon.smithi083 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.158:0/282274589' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:41.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:41.445 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:42.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:40.084211+0000 mgr.smithi083.dowdym (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:43.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.281+0000 7f6bb2af6700 1 -- 172.21.15.158:0/261303768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 msgr2=0x7f6bac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.281+0000 7f6bb2af6700 1 --2- 172.21.15.158:0/261303768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f6ba0009fb0 tx=0x7f6ba0009b30).stop 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.281+0000 7f6bb2af6700 1 -- 172.21.15.158:0/261303768 shutdown_connections 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.281+0000 7f6bb2af6700 1 --2- 172.21.15.158:0/261303768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.281+0000 7f6bb2af6700 1 -- 172.21.15.158:0/261303768 >> 172.21.15.158:0/261303768 conn(0x7f6bac0f0b70 msgr2=0x7f6bac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.282+0000 7f6bb2af6700 1 -- 172.21.15.158:0/261303768 shutdown_connections 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.282+0000 7f6bb2af6700 1 -- 172.21.15.158:0/261303768 wait complete. 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.282+0000 7f6bb2af6700 1 Processor -- start 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.282+0000 7f6bb2af6700 1 -- start start 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.282+0000 7f6bb2af6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.282+0000 7f6bb2af6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0f9a00 con 0x7f6bac0f5410 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb0892700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:43.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb0892700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57012/0 (socket says 172.21.15.158:57012) 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb0892700 1 -- 172.21.15.158:0/498091019 learned_addr learned my addr 172.21.15.158:0/498091019 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb0892700 1 -- 172.21.15.158:0/498091019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0005040 con 0x7f6bac0f5410 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb0892700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0fb580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6ba0005010 tx=0x7f6ba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ba0009320 con 0x7f6bac0f5410 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ba0013380 con 0x7f6bac0f5410 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac0f9ba0 con 0x7f6bac0f5410 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ba0014e50 con 0x7f6bac0f5410 2024-02-10T09:27:43.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.283+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac0fa030 con 0x7f6bac0f5410 2024-02-10T09:27:43.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.284+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ba0014420 con 0x7f6bac0f5410 2024-02-10T09:27:43.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.284+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac059100 con 0x7f6bac0f5410 2024-02-10T09:27:43.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.284+0000 7f6ba97fa700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6b9c02ff10 0x7f6b9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:43.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.284+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ba0040e30 con 0x7f6bac0f5410 2024-02-10T09:27:43.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.284+0000 7f6babfff700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6b9c02ff10 0x7f6b9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:43.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.285+0000 7f6babfff700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6b9c02ff10 0x7f6b9c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6b980096a0 tx=0x7f6b98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:43.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.289+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ba00095f0 con 0x7f6bac0f5410 2024-02-10T09:27:43.494 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.494+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6bac0fc0d0 con 0x7f6bac0f5410 2024-02-10T09:27:43.495 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.494+0000 7f6ba97fa700 1 -- 172.21.15.158:0/498091019 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ba001db70 con 0x7f6bac0f5410 2024-02-10T09:27:43.495 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:43.495 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:43.496 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6b9c02ff10 msgr2=0x7f6b9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:43.496 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6b9c02ff10 0x7f6b9c0323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6b980096a0 tx=0x7f6b98006b40).stop 2024-02-10T09:27:43.496 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 msgr2=0x7f6bac0fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:43.496 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0fb580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6ba0005010 tx=0x7f6ba000b040).stop 2024-02-10T09:27:43.496 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 shutdown_connections 2024-02-10T09:27:43.497 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6b9c02ff10 0x7f6b9c0323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:43.497 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 --2- 172.21.15.158:0/498091019 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6bac0f5410 0x7f6bac0fb580 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:43.497 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 >> 172.21.15.158:0/498091019 conn(0x7f6bac0f0b70 msgr2=0x7f6bac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:43.497 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 shutdown_connections 2024-02-10T09:27:43.497 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:43.496+0000 7f6bb2af6700 1 -- 172.21.15.158:0/498091019 wait complete. 2024-02-10T09:27:43.497 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:43.978 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:42.084500+0000 mgr.smithi083.dowdym (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:43.978 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:43.494850+0000 mon.smithi083 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.158:0/498091019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:45.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:45.145 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:45.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:44.046657+0000 mon.smithi083 (mon.0) 215 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:27:45.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:27:44.047980+0000 mgr.smithi083.dowdym (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi083 on smithi083 2024-02-10T09:27:46.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:46 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:44.084714+0000 mgr.smithi083.dowdym (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:47.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 -- 172.21.15.158:0/1939029058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 msgr2=0x7f9b680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:47.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 --2- 172.21.15.158:0/1939029058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f9b64009fb0 tx=0x7f9b64009b30).stop 2024-02-10T09:27:47.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 -- 172.21.15.158:0/1939029058 shutdown_connections 2024-02-10T09:27:47.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 --2- 172.21.15.158:0/1939029058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 -- 172.21.15.158:0/1939029058 >> 172.21.15.158:0/1939029058 conn(0x7f9b680f09a0 msgr2=0x7f9b680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 -- 172.21.15.158:0/1939029058 shutdown_connections 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.074+0000 7f9b6f09d700 1 -- 172.21.15.158:0/1939029058 wait complete. 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6f09d700 1 Processor -- start 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6f09d700 1 -- start start 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6f09d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:47.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6f09d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b680fb6f0 con 0x7f9b680f51a0 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6e09b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6e09b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57024/0 (socket says 172.21.15.158:57024) 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6e09b700 1 -- 172.21.15.158:0/3300998620 learned_addr learned my addr 172.21.15.158:0/3300998620 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.075+0000 7f9b6e09b700 1 -- 172.21.15.158:0/3300998620 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64005040 con 0x7f9b680f51a0 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.076+0000 7f9b6e09b700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680fd270 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f9b64000c00 tx=0x7f9b6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.076+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b640095f0 con 0x7f9b680f51a0 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.076+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b64007e90 con 0x7f9b680f51a0 2024-02-10T09:27:47.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.076+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b680fb890 con 0x7f9b680f51a0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.076+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b64014db0 con 0x7f9b680f51a0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.076+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b680fbd20 con 0x7f9b680f51a0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.077+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b64025070 con 0x7f9b680f51a0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.077+0000 7f9b5effd700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b5402ff60 0x7f9b54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.077+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b64040890 con 0x7f9b680f51a0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.077+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b68059100 con 0x7f9b680f51a0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.077+0000 7f9b6d89a700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b5402ff60 0x7f9b54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:47.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.077+0000 7f9b6d89a700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b5402ff60 0x7f9b54032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9b580096a0 tx=0x7f9b58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:47.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.081+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b64026350 con 0x7f9b680f51a0 2024-02-10T09:27:47.286 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.286+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b680fc420 con 0x7f9b680f51a0 2024-02-10T09:27:47.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.286+0000 7f9b5effd700 1 -- 172.21.15.158:0/3300998620 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b64025300 con 0x7f9b680f51a0 2024-02-10T09:27:47.287 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:47.287 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b5402ff60 msgr2=0x7f9b54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b5402ff60 0x7f9b54032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9b580096a0 tx=0x7f9b58006b40).stop 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 msgr2=0x7f9b680fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680fd270 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f9b64000c00 tx=0x7f9b6400b040).stop 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 shutdown_connections 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b5402ff60 0x7f9b54032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 --2- 172.21.15.158:0/3300998620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b680f51a0 0x7f9b680fd270 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 >> 172.21.15.158:0/3300998620 conn(0x7f9b680f09a0 msgr2=0x7f9b6805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 shutdown_connections 2024-02-10T09:27:47.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:47.288+0000 7f9b6f09d700 1 -- 172.21.15.158:0/3300998620 wait complete. 2024-02-10T09:27:47.290 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:48.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:46.085033+0000 mgr.smithi083.dowdym (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:48.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:47.286913+0000 mon.smithi083 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.158:0/3300998620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:48.978 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:48.978 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:50.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:50 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:48.085364+0000 mgr.smithi083.dowdym (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:50.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 -- 172.21.15.158:0/3290775774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 msgr2=0x7efdb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:50.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 --2- 172.21.15.158:0/3290775774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb00f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7efda0009fb0 tx=0x7efda0009b30).stop 2024-02-10T09:27:50.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 -- 172.21.15.158:0/3290775774 shutdown_connections 2024-02-10T09:27:50.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 --2- 172.21.15.158:0/3290775774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb00f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:50.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 -- 172.21.15.158:0/3290775774 >> 172.21.15.158:0/3290775774 conn(0x7efdb00f1c70 msgr2=0x7efdb00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:50.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 -- 172.21.15.158:0/3290775774 shutdown_connections 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.868+0000 7efdb5e75700 1 -- 172.21.15.158:0/3290775774 wait complete. 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.869+0000 7efdb5e75700 1 Processor -- start 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.869+0000 7efdb5e75700 1 -- start start 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.869+0000 7efdb5e75700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.869+0000 7efdb5e75700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0069e20 con 0x7efdb00f6510 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdaf7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdaf7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33942/0 (socket says 172.21.15.158:33942) 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdaf7fe700 1 -- 172.21.15.158:0/1542030455 learned_addr learned my addr 172.21.15.158:0/1542030455 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdaf7fe700 1 -- 172.21.15.158:0/1542030455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0005040 con 0x7efdb00f6510 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdaf7fe700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb006b9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7efda0005010 tx=0x7efda000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda0009320 con 0x7efdb00f6510 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb0069fc0 con 0x7efdb00f6510 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efda0013380 con 0x7efdb00f6510 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda0014e50 con 0x7efdb00f6510 2024-02-10T09:27:50.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.870+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb006a450 con 0x7efdb00f6510 2024-02-10T09:27:50.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.871+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efda00095f0 con 0x7efdb00f6510 2024-02-10T09:27:50.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.871+0000 7efd97fff700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7efd9802ff10 0x7efd980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:50.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.871+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efda0040fa0 con 0x7efdb00f6510 2024-02-10T09:27:50.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.871+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb0059100 con 0x7efdb00f6510 2024-02-10T09:27:50.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.871+0000 7efdaeffd700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7efd9802ff10 0x7efd980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:50.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.872+0000 7efdaeffd700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7efd9802ff10 0x7efd980323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efda40096a0 tx=0x7efda4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:50.876 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:50.876+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efda0014420 con 0x7efdb00f6510 2024-02-10T09:27:51.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.081+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efdb006ab50 con 0x7efdb00f6510 2024-02-10T09:27:51.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.081+0000 7efd97fff700 1 -- 172.21.15.158:0/1542030455 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efda0010230 con 0x7efdb00f6510 2024-02-10T09:27:51.082 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:51.082 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:51.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7efd9802ff10 msgr2=0x7efd980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:51.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7efd9802ff10 0x7efd980323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efda40096a0 tx=0x7efda4006b40).stop 2024-02-10T09:27:51.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 msgr2=0x7efdb006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:51.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb006b9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7efda0005010 tx=0x7efda000b040).stop 2024-02-10T09:27:51.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 shutdown_connections 2024-02-10T09:27:51.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7efd9802ff10 0x7efd980323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:51.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 --2- 172.21.15.158:0/1542030455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb00f6510 0x7efdb006b9a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:51.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 >> 172.21.15.158:0/1542030455 conn(0x7efdb00f1c70 msgr2=0x7efdb00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:51.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 shutdown_connections 2024-02-10T09:27:51.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:51.083+0000 7efdb5e75700 1 -- 172.21.15.158:0/1542030455 wait complete. 2024-02-10T09:27:51.084 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:52.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:52 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:50.085650+0000 mgr.smithi083.dowdym (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:52.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:52 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:51.081857+0000 mon.smithi083 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.158:0/1542030455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:52.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:52.739 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:54.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:54 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:52.085939+0000 mgr.smithi083.dowdym (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:54.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 -- 172.21.15.158:0/4082069498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 msgr2=0x7ff81c0f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 --2- 172.21.15.158:0/4082069498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c0f4fa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff810009fb0 tx=0x7ff810009b30).stop 2024-02-10T09:27:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 -- 172.21.15.158:0/4082069498 shutdown_connections 2024-02-10T09:27:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 --2- 172.21.15.158:0/4082069498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c0f4fa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 -- 172.21.15.158:0/4082069498 >> 172.21.15.158:0/4082069498 conn(0x7ff81c0f02e0 msgr2=0x7ff81c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 -- 172.21.15.158:0/4082069498 shutdown_connections 2024-02-10T09:27:54.591 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.589+0000 7ff8214bb700 1 -- 172.21.15.158:0/4082069498 wait complete. 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff8214bb700 1 Processor -- start 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff8214bb700 1 -- start start 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff8214bb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff8214bb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c106cc0 con 0x7ff81c0f4b80 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff81affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff81affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33954/0 (socket says 172.21.15.158:33954) 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.590+0000 7ff81affd700 1 -- 172.21.15.158:0/1384203046 learned_addr learned my addr 172.21.15.158:0/1384203046 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff81affd700 1 -- 172.21.15.158:0/1384203046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff810005040 con 0x7ff81c0f4b80 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff81affd700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c1067d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff810005010 tx=0x7ff81000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff810009320 con 0x7ff81c0f4b80 2024-02-10T09:27:54.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c106ec0 con 0x7ff81c0f4b80 2024-02-10T09:27:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff810013380 con 0x7ff81c0f4b80 2024-02-10T09:27:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff810014e50 con 0x7ff81c0f4b80 2024-02-10T09:27:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.591+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c107350 con 0x7ff81c0f4b80 2024-02-10T09:27:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.592+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff810014420 con 0x7ff81c0f4b80 2024-02-10T09:27:54.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.592+0000 7ff8037fe700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff80402ff10 0x7ff8040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.592+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81c045ad0 con 0x7ff81c0f4b80 2024-02-10T09:27:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.592+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff810040f40 con 0x7ff81c0f4b80 2024-02-10T09:27:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.592+0000 7ff81a7fc700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff80402ff10 0x7ff8040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:54.595 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.593+0000 7ff81a7fc700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff80402ff10 0x7ff8040323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff80c0096a0 tx=0x7ff80c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:54.598 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.597+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8100095f0 con 0x7ff81c0f4b80 2024-02-10T09:27:54.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.802+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff81c0fa710 con 0x7ff81c0f4b80 2024-02-10T09:27:54.804 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.803+0000 7ff8037fe700 1 -- 172.21.15.158:0/1384203046 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff81001db70 con 0x7ff81c0f4b80 2024-02-10T09:27:54.804 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:54.805 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff80402ff10 msgr2=0x7ff8040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff80402ff10 0x7ff8040323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7ff80c0096a0 tx=0x7ff80c006b40).stop 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 msgr2=0x7ff81c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c1067d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff810005010 tx=0x7ff81000b040).stop 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 shutdown_connections 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff80402ff10 0x7ff8040323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 --2- 172.21.15.158:0/1384203046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff81c0f4b80 0x7ff81c1067d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 >> 172.21.15.158:0/1384203046 conn(0x7ff81c0f02e0 msgr2=0x7ff81c0f1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 shutdown_connections 2024-02-10T09:27:54.806 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:54.804+0000 7ff8214bb700 1 -- 172.21.15.158:0/1384203046 wait complete. 2024-02-10T09:27:54.807 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:56.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:54.086266+0000 mgr.smithi083.dowdym (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:56.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:54.804150+0000 mon.smithi083 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.158:0/1384203046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:27:56.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:27:56.521 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:27:58.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:56.086666+0000 mgr.smithi083.dowdym (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:27:58.471 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- 172.21.15.158:0/3159498398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 msgr2=0x7f790c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:58.471 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 --2- 172.21.15.158:0/3159498398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c0f8f20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f7908009fb0 tx=0x7f7908009b30).stop 2024-02-10T09:27:58.471 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- 172.21.15.158:0/3159498398 shutdown_connections 2024-02-10T09:27:58.471 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 --2- 172.21.15.158:0/3159498398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c0f8f20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- 172.21.15.158:0/3159498398 >> 172.21.15.158:0/3159498398 conn(0x7f790c0f09a0 msgr2=0x7f790c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- 172.21.15.158:0/3159498398 shutdown_connections 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- 172.21.15.158:0/3159498398 wait complete. 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 Processor -- start 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- start start 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.459+0000 7f79125bb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c106af0 con 0x7f790c0601e0 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f79115b9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f79115b9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33976/0 (socket says 172.21.15.158:33976) 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f79115b9700 1 -- 172.21.15.158:0/2684772700 learned_addr learned my addr 172.21.15.158:0/2684772700 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f79115b9700 1 -- 172.21.15.158:0/2684772700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7908005040 con 0x7f790c0601e0 2024-02-10T09:27:58.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f79115b9700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c106600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7908005010 tx=0x7f790800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7908009320 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c106cf0 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7908013380 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.460+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7908014e50 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.461+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c107180 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.462+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79080095f0 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.462+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c045ad0 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.462+0000 7f7901ffb700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f78f802ff10 0x7f78f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.462+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7908040e70 con 0x7f790c0601e0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.462+0000 7f7910db8700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f78f802ff10 0x7f78f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.462+0000 7f7910db8700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f78f802ff10 0x7f78f80323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f78fc0096a0 tx=0x7f78fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:27:58.473 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.467+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7908014420 con 0x7f790c0601e0 2024-02-10T09:27:58.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.672+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f790c108030 con 0x7f790c0601e0 2024-02-10T09:27:58.674 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.672+0000 7f7901ffb700 1 -- 172.21.15.158:0/2684772700 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7908010230 con 0x7f790c0601e0 2024-02-10T09:27:58.674 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:27:58.674 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f78f802ff10 msgr2=0x7f78f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f78f802ff10 0x7f78f80323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f78fc0096a0 tx=0x7f78fc006b40).stop 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 msgr2=0x7f790c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c106600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f7908005010 tx=0x7f790800b040).stop 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 shutdown_connections 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f78f802ff10 0x7f78f80323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 --2- 172.21.15.158:0/2684772700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f790c0601e0 0x7f790c106600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 >> 172.21.15.158:0/2684772700 conn(0x7f790c0f09a0 msgr2=0x7f790c0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 shutdown_connections 2024-02-10T09:27:58.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:27:58.674+0000 7f79125bb700 1 -- 172.21.15.158:0/2684772700 wait complete. 2024-02-10T09:27:58.677 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:27:59.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:27:58.673913+0000 mon.smithi083 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.158:0/2684772700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:00.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:27:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:27:58.086995+0000 mgr.smithi083.dowdym (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:00.396 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:00.396 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:02.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:00.087260+0000 mgr.smithi083.dowdym (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:02.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- 172.21.15.158:0/4203615203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 msgr2=0x7f1bb00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 --2- 172.21.15.158:0/4203615203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb00f66c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f1ba4009fb0 tx=0x7f1ba4009b30).stop 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- 172.21.15.158:0/4203615203 shutdown_connections 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 --2- 172.21.15.158:0/4203615203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb00f66c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- 172.21.15.158:0/4203615203 >> 172.21.15.158:0/4203615203 conn(0x7f1bb00f1aa0 msgr2=0x7f1bb00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- 172.21.15.158:0/4203615203 shutdown_connections 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- 172.21.15.158:0/4203615203 wait complete. 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 Processor -- start 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- start start 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.242+0000 7f1baf59e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0106ac0 con 0x7f1bb00f62a0 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1bae59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:02.245 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1bae59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:50106/0 (socket says 172.21.15.158:50106) 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1bae59c700 1 -- 172.21.15.158:0/691122769 learned_addr learned my addr 172.21.15.158:0/691122769 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1bae59c700 1 -- 172.21.15.158:0/691122769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba4005040 con 0x7f1bb00f62a0 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1bae59c700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb01065d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f1ba4009830 tx=0x7f1ba400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ba4009320 con 0x7f1bb00f62a0 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ba4013380 con 0x7f1bb00f62a0 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb0106cc0 con 0x7f1bb00f62a0 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ba4014420 con 0x7f1bb00f62a0 2024-02-10T09:28:02.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.243+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb0107150 con 0x7f1bb00f62a0 2024-02-10T09:28:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.244+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ba401e420 con 0x7f1bb00f62a0 2024-02-10T09:28:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.244+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb0045ad0 con 0x7f1bb00f62a0 2024-02-10T09:28:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.244+0000 7f1b9effd700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b9802ff60 0x7f1b98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.244+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ba4040e20 con 0x7f1bb00f62a0 2024-02-10T09:28:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.244+0000 7f1badd9b700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b9802ff60 0x7f1b98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:02.247 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.245+0000 7f1badd9b700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b9802ff60 0x7f1b98032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1ba00096a0 tx=0x7f1ba0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:02.250 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.249+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ba401c970 con 0x7f1bb00f62a0 2024-02-10T09:28:02.456 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.454+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bb00fbe30 con 0x7f1bb00f62a0 2024-02-10T09:28:02.456 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.455+0000 7f1b9effd700 1 -- 172.21.15.158:0/691122769 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ba402c2d0 con 0x7f1bb00f62a0 2024-02-10T09:28:02.456 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:02.456 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:02.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b9802ff60 msgr2=0x7f1b98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:02.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b9802ff60 0x7f1b98032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1ba00096a0 tx=0x7f1ba0006b40).stop 2024-02-10T09:28:02.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 msgr2=0x7f1bb01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:02.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb01065d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f1ba4009830 tx=0x7f1ba400b040).stop 2024-02-10T09:28:02.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 shutdown_connections 2024-02-10T09:28:02.459 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b9802ff60 0x7f1b98032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:02.459 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 --2- 172.21.15.158:0/691122769 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1bb00f62a0 0x7f1bb01065d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:02.459 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 >> 172.21.15.158:0/691122769 conn(0x7f1bb00f1aa0 msgr2=0x7f1bb00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:02.459 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 shutdown_connections 2024-02-10T09:28:02.459 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:02.457+0000 7f1baf59e700 1 -- 172.21.15.158:0/691122769 wait complete. 2024-02-10T09:28:02.459 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:03.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:02 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:02.456350+0000 mon.smithi083 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.158:0/691122769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:04.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:02.087545+0000 mgr.smithi083.dowdym (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:04.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:04.167 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:06.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:04.087819+0000 mgr.smithi083.dowdym (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:06.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.156+0000 7f56e8ea0700 1 -- 172.21.15.158:0/110163181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 msgr2=0x7f56e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:06.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.156+0000 7f56e8ea0700 1 --2- 172.21.15.158:0/110163181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e40f6930 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f56d8009fb0 tx=0x7f56d8009b30).stop 2024-02-10T09:28:06.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.156+0000 7f56e8ea0700 1 -- 172.21.15.158:0/110163181 shutdown_connections 2024-02-10T09:28:06.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.156+0000 7f56e8ea0700 1 --2- 172.21.15.158:0/110163181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e40f6930 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:06.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 -- 172.21.15.158:0/110163181 >> 172.21.15.158:0/110163181 conn(0x7f56e40f1c70 msgr2=0x7f56e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:06.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 -- 172.21.15.158:0/110163181 shutdown_connections 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 -- 172.21.15.158:0/110163181 wait complete. 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 Processor -- start 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 -- start start 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e4108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e8ea0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4108ef0 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e4108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e4108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:50126/0 (socket says 172.21.15.158:50126) 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.157+0000 7f56e259c700 1 -- 172.21.15.158:0/2781120715 learned_addr learned my addr 172.21.15.158:0/2781120715 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56e259c700 1 -- 172.21.15.158:0/2781120715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d8005040 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56e259c700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e4108a00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f56d8005010 tx=0x7f56d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d8009320 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56d8013380 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d8014e50 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e41090f0 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.158+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e4109580 con 0x7f56e40f6510 2024-02-10T09:28:06.175 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.159+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56d80095f0 con 0x7f56e40f6510 2024-02-10T09:28:06.176 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.159+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e4045ad0 con 0x7f56e40f6510 2024-02-10T09:28:06.176 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.159+0000 7f56d2ffd700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56cc02ff10 0x7f56cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:06.176 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.159+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56d8040e60 con 0x7f56e40f6510 2024-02-10T09:28:06.176 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.159+0000 7f56e1d9b700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56cc02ff10 0x7f56cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:06.176 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.160+0000 7f56e1d9b700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56cc02ff10 0x7f56cc0323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f56d40096a0 tx=0x7f56d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:06.176 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.164+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56d8014420 con 0x7f56e40f6510 2024-02-10T09:28:06.372 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.371+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56e410a430 con 0x7f56e40f6510 2024-02-10T09:28:06.373 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.372+0000 7f56d2ffd700 1 -- 172.21.15.158:0/2781120715 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56d801d980 con 0x7f56e40f6510 2024-02-10T09:28:06.373 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:06.373 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56cc02ff10 msgr2=0x7f56cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56cc02ff10 0x7f56cc0323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f56d40096a0 tx=0x7f56d4006b40).stop 2024-02-10T09:28:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 msgr2=0x7f56e4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e4108a00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f56d8005010 tx=0x7f56d800b040).stop 2024-02-10T09:28:06.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 shutdown_connections 2024-02-10T09:28:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56cc02ff10 0x7f56cc0323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 --2- 172.21.15.158:0/2781120715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56e40f6510 0x7f56e4108a00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 >> 172.21.15.158:0/2781120715 conn(0x7f56e40f1c70 msgr2=0x7f56e40f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 shutdown_connections 2024-02-10T09:28:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:06.374+0000 7f56e8ea0700 1 -- 172.21.15.158:0/2781120715 wait complete. 2024-02-10T09:28:06.376 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:07.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:06 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:06.373164+0000 mon.smithi083 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.158:0/2781120715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:08.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:06.088095+0000 mgr.smithi083.dowdym (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:08.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:08.125 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:10.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:08.088426+0000 mgr.smithi083.dowdym (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.077+0000 7f131c45f700 1 -- 172.21.15.158:0/4212408850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 msgr2=0x7f13140f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.077+0000 7f131c45f700 1 --2- 172.21.15.158:0/4212408850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13140f68a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1304009fb0 tx=0x7f1304009b30).stop 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 -- 172.21.15.158:0/4212408850 shutdown_connections 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 --2- 172.21.15.158:0/4212408850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13140f68a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 -- 172.21.15.158:0/4212408850 >> 172.21.15.158:0/4212408850 conn(0x7f13140f1c00 msgr2=0x7f13140f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 -- 172.21.15.158:0/4212408850 shutdown_connections 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 -- 172.21.15.158:0/4212408850 wait complete. 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 Processor -- start 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 -- start start 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13141067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:10.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131c45f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314106ce0 con 0x7f13140f6480 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131a1fb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13141067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131a1fb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13141067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33008/0 (socket says 172.21.15.158:33008) 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.078+0000 7f131a1fb700 1 -- 172.21.15.158:0/2789663092 learned_addr learned my addr 172.21.15.158:0/2789663092 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f131a1fb700 1 -- 172.21.15.158:0/2789663092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304005040 con 0x7f13140f6480 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f131a1fb700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13141067f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f13040089f0 tx=0x7f130400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1304009320 con 0x7f13140f6480 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1314106ee0 con 0x7f13140f6480 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1304013380 con 0x7f13140f6480 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f130401ed80 con 0x7f13140f6480 2024-02-10T09:28:10.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.079+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1314107370 con 0x7f13140f6480 2024-02-10T09:28:10.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.080+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1304014420 con 0x7f13140f6480 2024-02-10T09:28:10.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.080+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1314045ad0 con 0x7f13140f6480 2024-02-10T09:28:10.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.080+0000 7f130affd700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f130002ff60 0x7f1300032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:10.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.080+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1304040fe0 con 0x7f13140f6480 2024-02-10T09:28:10.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.080+0000 7f13199fa700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f130002ff60 0x7f1300032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:10.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.081+0000 7f13199fa700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f130002ff60 0x7f1300032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f13100096a0 tx=0x7f1310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:10.086 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.085+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13040095f0 con 0x7f13140f6480 2024-02-10T09:28:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.290+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13140fc090 con 0x7f13140f6480 2024-02-10T09:28:10.292 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.291+0000 7f130affd700 1 -- 172.21.15.158:0/2789663092 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f130401cb10 con 0x7f13140f6480 2024-02-10T09:28:10.293 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:10.293 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f130002ff60 msgr2=0x7f1300032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f130002ff60 0x7f1300032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f13100096a0 tx=0x7f1310006b40).stop 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 msgr2=0x7f13141067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13141067f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f13040089f0 tx=0x7f130400b040).stop 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 shutdown_connections 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f130002ff60 0x7f1300032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 --2- 172.21.15.158:0/2789663092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13140f6480 0x7f13141067f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 >> 172.21.15.158:0/2789663092 conn(0x7f13140f1c00 msgr2=0x7f13140f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 shutdown_connections 2024-02-10T09:28:10.295 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:10.293+0000 7f131c45f700 1 -- 172.21.15.158:0/2789663092 wait complete. 2024-02-10T09:28:10.296 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:11.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:10.292691+0000 mon.smithi083 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.158:0/2789663092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:12.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:10.088693+0000 mgr.smithi083.dowdym (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:12.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:12.056 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 -- 172.21.15.158:0/440652257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 msgr2=0x7fe7bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 --2- 172.21.15.158:0/440652257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc0f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fe7b0009fb0 tx=0x7fe7b0009b30).stop 2024-02-10T09:28:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 -- 172.21.15.158:0/440652257 shutdown_connections 2024-02-10T09:28:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 --2- 172.21.15.158:0/440652257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc0f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 -- 172.21.15.158:0/440652257 >> 172.21.15.158:0/440652257 conn(0x7fe7bc0f0b70 msgr2=0x7fe7bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:14.004 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 -- 172.21.15.158:0/440652257 shutdown_connections 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.001+0000 7fe7c4431700 1 -- 172.21.15.158:0/440652257 wait complete. 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c4431700 1 Processor -- start 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c4431700 1 -- start start 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c4431700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c4431700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc106ca0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c21cd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c21cd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33034/0 (socket says 172.21.15.158:33034) 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.002+0000 7fe7c21cd700 1 -- 172.21.15.158:0/3921317365 learned_addr learned my addr 172.21.15.158:0/3921317365 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7c21cd700 1 -- 172.21.15.158:0/3921317365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b0005040 con 0x7fe7bc0f5410 2024-02-10T09:28:14.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7c21cd700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc1067b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fe7b0005010 tx=0x7fe7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:14.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b0009320 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc106ea0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc107330 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7b0013380 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.003+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b0014df0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.004+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7b00095f0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.004+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a4004fa0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.004+0000 7fe7aeffd700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7a802ff10 0x7fe7a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.004+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7b0040df0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.005+0000 7fe7c19cc700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7a802ff10 0x7fe7a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:14.007 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.005+0000 7fe7c19cc700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7a802ff10 0x7fe7a80323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe7b80096a0 tx=0x7fe7b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:14.010 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.009+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7b0014420 con 0x7fe7bc0f5410 2024-02-10T09:28:14.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:12.088947+0000 mgr.smithi083.dowdym (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:14.216 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.215+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7a4005670 con 0x7fe7bc0f5410 2024-02-10T09:28:14.217 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.216+0000 7fe7aeffd700 1 -- 172.21.15.158:0/3921317365 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe7b001d9d0 con 0x7fe7bc0f5410 2024-02-10T09:28:14.217 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:14.217 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7a802ff10 msgr2=0x7fe7a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7a802ff10 0x7fe7a80323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe7b80096a0 tx=0x7fe7b8006b40).stop 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 msgr2=0x7fe7bc1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc1067b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fe7b0005010 tx=0x7fe7b000b040).stop 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 shutdown_connections 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7a802ff10 0x7fe7a80323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 --2- 172.21.15.158:0/3921317365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe7bc0f5410 0x7fe7bc1067b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 >> 172.21.15.158:0/3921317365 conn(0x7fe7bc0f0b70 msgr2=0x7fe7bc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 shutdown_connections 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:14.217+0000 7fe7c4431700 1 -- 172.21.15.158:0/3921317365 wait complete. 2024-02-10T09:28:14.219 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:15.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:14.171528+0000 mon.smithi083 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:28:15.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:14.179522+0000 mon.smithi083 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:28:15.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:14.217009+0000 mon.smithi083 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.158:0/3921317365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:15.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:15.940 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:16.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:14.089290+0000 mgr.smithi083.dowdym (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:17.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 -- 172.21.15.158:0/1639876048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 msgr2=0x7fb9c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:17.913 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 --2- 172.21.15.158:0/1639876048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb9b8009fb0 tx=0x7fb9b8009b30).stop 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 -- 172.21.15.158:0/1639876048 shutdown_connections 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 --2- 172.21.15.158:0/1639876048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 -- 172.21.15.158:0/1639876048 >> 172.21.15.158:0/1639876048 conn(0x7fb9c80f0b70 msgr2=0x7fb9c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 -- 172.21.15.158:0/1639876048 shutdown_connections 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.911+0000 7fb9d011e700 1 -- 172.21.15.158:0/1639876048 wait complete. 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9d011e700 1 Processor -- start 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9d011e700 1 -- start start 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9d011e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:17.914 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9d011e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80fb960 con 0x7fb9c80f5410 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9cdeba700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9cdeba700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33050/0 (socket says 172.21.15.158:33050) 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.912+0000 7fb9cdeba700 1 -- 172.21.15.158:0/414559046 learned_addr learned my addr 172.21.15.158:0/414559046 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9cdeba700 1 -- 172.21.15.158:0/414559046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8005040 con 0x7fb9c80f5410 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9cdeba700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80fd4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fb9b8005010 tx=0x7fb9b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9b8009320 con 0x7fb9c80f5410 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c80fbb00 con 0x7fb9c80f5410 2024-02-10T09:28:17.916 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9b8013380 con 0x7fb9c80f5410 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c80fbf90 con 0x7fb9c80f5410 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.913+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9b8014e50 con 0x7fb9c80f5410 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.914+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9b80095f0 con 0x7fb9c80f5410 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.914+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c8059100 con 0x7fb9c80f5410 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.914+0000 7fb9be7fc700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9b402ff10 0x7fb9b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.914+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9b8040f00 con 0x7fb9c80f5410 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.914+0000 7fb9cd6b9700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9b402ff10 0x7fb9b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:17.917 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.915+0000 7fb9cd6b9700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9b402ff10 0x7fb9b40323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb9c40096a0 tx=0x7fb9c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:17.919 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:17.918+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9b8014420 con 0x7fb9c80f5410 2024-02-10T09:28:18.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:16.089667+0000 mgr.smithi083.dowdym (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:18.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.123+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9c8045ad0 con 0x7fb9c80f5410 2024-02-10T09:28:18.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.124+0000 7fb9be7fc700 1 -- 172.21.15.158:0/414559046 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9b801d9d0 con 0x7fb9c80f5410 2024-02-10T09:28:18.125 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:18.125 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9b402ff10 msgr2=0x7fb9b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9b402ff10 0x7fb9b40323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb9c40096a0 tx=0x7fb9c4006b40).stop 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 msgr2=0x7fb9c80fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80fd4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fb9b8005010 tx=0x7fb9b800b040).stop 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 shutdown_connections 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9b402ff10 0x7fb9b40323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 --2- 172.21.15.158:0/414559046 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9c80f5410 0x7fb9c80fd4e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 >> 172.21.15.158:0/414559046 conn(0x7fb9c80f0b70 msgr2=0x7fb9c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 shutdown_connections 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:18.125+0000 7fb9d011e700 1 -- 172.21.15.158:0/414559046 wait complete. 2024-02-10T09:28:18.127 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:19.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:18 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:18.124934+0000 mon.smithi083 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.158:0/414559046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:19.669 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:19.670 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:20.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:18.090026+0000 mgr.smithi083.dowdym (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 -- 172.21.15.158:0/1861931917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 msgr2=0x7fdabc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 --2- 172.21.15.158:0/1861931917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc0f9150 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fdaac009fb0 tx=0x7fdaac009b30).stop 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 -- 172.21.15.158:0/1861931917 shutdown_connections 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 --2- 172.21.15.158:0/1861931917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc0f9150 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 -- 172.21.15.158:0/1861931917 >> 172.21.15.158:0/1861931917 conn(0x7fdabc0f0b70 msgr2=0x7fdabc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 -- 172.21.15.158:0/1861931917 shutdown_connections 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 -- 172.21.15.158:0/1861931917 wait complete. 2024-02-10T09:28:21.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 Processor -- start 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.448+0000 7fdac2048700 1 -- start start 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdac2048700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdac2048700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdabc106cb0 con 0x7fdabc060220 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdabb7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdabb7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:51030/0 (socket says 172.21.15.158:51030) 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdabb7fe700 1 -- 172.21.15.158:0/1108893838 learned_addr learned my addr 172.21.15.158:0/1108893838 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdabb7fe700 1 -- 172.21.15.158:0/1108893838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaac005040 con 0x7fdabc060220 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdabb7fe700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc1067c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fdaac0089f0 tx=0x7fdaac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaac009320 con 0x7fdabc060220 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdabc106eb0 con 0x7fdabc060220 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdaac013380 con 0x7fdabc060220 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.449+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaac014e20 con 0x7fdabc060220 2024-02-10T09:28:21.452 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.450+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdabc107340 con 0x7fdabc060220 2024-02-10T09:28:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.450+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaac014420 con 0x7fdabc060220 2024-02-10T09:28:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.451+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdabc045ad0 con 0x7fdabc060220 2024-02-10T09:28:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.451+0000 7fdaa3fff700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdaa402ff10 0x7fdaa40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.451+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaac040f00 con 0x7fdabc060220 2024-02-10T09:28:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.451+0000 7fdabaffd700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdaa402ff10 0x7fdaa40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:21.454 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.451+0000 7fdabaffd700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdaa402ff10 0x7fdaa40323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fdab00096a0 tx=0x7fdab0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:21.456 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.455+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaac01dd50 con 0x7fdabc060220 2024-02-10T09:28:21.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.661+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdabc1081f0 con 0x7fdabc060220 2024-02-10T09:28:21.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.662+0000 7fdaa3fff700 1 -- 172.21.15.158:0/1108893838 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaac01db70 con 0x7fdabc060220 2024-02-10T09:28:21.663 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:21.663 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdaa402ff10 msgr2=0x7fdaa40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdaa402ff10 0x7fdaa40323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fdab00096a0 tx=0x7fdab0006b40).stop 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 msgr2=0x7fdabc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc1067c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fdaac0089f0 tx=0x7fdaac00b040).stop 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 shutdown_connections 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdaa402ff10 0x7fdaa40323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 --2- 172.21.15.158:0/1108893838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdabc060220 0x7fdabc1067c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 >> 172.21.15.158:0/1108893838 conn(0x7fdabc0f0b70 msgr2=0x7fdabc0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.663+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 shutdown_connections 2024-02-10T09:28:21.665 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:21.664+0000 7fdac2048700 1 -- 172.21.15.158:0/1108893838 wait complete. 2024-02-10T09:28:21.666 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:22.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:20.090413+0000 mgr.smithi083.dowdym (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:22.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:21.663017+0000 mon.smithi083 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.158:0/1108893838' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:23.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:23.299 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:24.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:22.090748+0000 mgr.smithi083.dowdym (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 -- 172.21.15.158:0/2813163852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 msgr2=0x7f68940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 --2- 172.21.15.158:0/2813163852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68940f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f687c009fb0 tx=0x7f687c009b30).stop 2024-02-10T09:28:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 -- 172.21.15.158:0/2813163852 shutdown_connections 2024-02-10T09:28:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 --2- 172.21.15.158:0/2813163852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68940f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:25.105 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 -- 172.21.15.158:0/2813163852 >> 172.21.15.158:0/2813163852 conn(0x7f68940f0b70 msgr2=0x7f68940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 -- 172.21.15.158:0/2813163852 shutdown_connections 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 -- 172.21.15.158:0/2813163852 wait complete. 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 Processor -- start 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.103+0000 7f689a3eb700 1 -- start start 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f689a3eb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68941067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f689a3eb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6894106ca0 con 0x7f68940f5410 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6893fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68941067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6893fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68941067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:51048/0 (socket says 172.21.15.158:51048) 2024-02-10T09:28:25.106 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6893fff700 1 -- 172.21.15.158:0/3196837615 learned_addr learned my addr 172.21.15.158:0/3196837615 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:25.107 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6893fff700 1 -- 172.21.15.158:0/3196837615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c005040 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6893fff700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68941067b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f687c005010 tx=0x7f687c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f687c009320 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6894106ea0 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f687c013380 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f687c014e50 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.104+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6894107330 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.105+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f687c0095f0 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.105+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6894045ad0 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.105+0000 7f6890ff9700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f688002ff10 0x7f68800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.105+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f687c040fa0 con 0x7f68940f5410 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.106+0000 7f68937fe700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f688002ff10 0x7f68800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:25.108 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.106+0000 7f68937fe700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f688002ff10 0x7f68800323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f68840096a0 tx=0x7f6884006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:25.111 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.110+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f687c014420 con 0x7f68940f5410 2024-02-10T09:28:25.314 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.313+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68940fafa0 con 0x7f68940f5410 2024-02-10T09:28:25.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.314+0000 7f6890ff9700 1 -- 172.21.15.158:0/3196837615 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f687c010230 con 0x7f68940f5410 2024-02-10T09:28:25.316 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:25.316 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.315+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f688002ff10 msgr2=0x7f68800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.315+0000 7f689a3eb700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f688002ff10 0x7f68800323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f68840096a0 tx=0x7f6884006b40).stop 2024-02-10T09:28:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 msgr2=0x7f68941067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68941067b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f687c005010 tx=0x7f687c00b040).stop 2024-02-10T09:28:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 shutdown_connections 2024-02-10T09:28:25.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f688002ff10 0x7f68800323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 --2- 172.21.15.158:0/3196837615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f68940f5410 0x7f68941067b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 >> 172.21.15.158:0/3196837615 conn(0x7f68940f0b70 msgr2=0x7f689405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 shutdown_connections 2024-02-10T09:28:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:25.316+0000 7f689a3eb700 1 -- 172.21.15.158:0/3196837615 wait complete. 2024-02-10T09:28:25.318 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:26.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:24.091034+0000 mgr.smithi083.dowdym (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:26.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:25.315129+0000 mon.smithi083 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.158:0/3196837615' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:27.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:27.000 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:28.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:26.091432+0000 mgr.smithi083.dowdym (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:28.811 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 -- 172.21.15.158:0/428029433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 msgr2=0x7fa8680f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 --2- 172.21.15.158:0/428029433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa8680f9190 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fa864009fb0 tx=0x7fa864009b30).stop 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 -- 172.21.15.158:0/428029433 shutdown_connections 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 --2- 172.21.15.158:0/428029433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa8680f9190 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 -- 172.21.15.158:0/428029433 >> 172.21.15.158:0/428029433 conn(0x7fa8680f0b70 msgr2=0x7fa8680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 -- 172.21.15.158:0/428029433 shutdown_connections 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.808+0000 7fa86f8db700 1 -- 172.21.15.158:0/428029433 wait complete. 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86f8db700 1 Processor -- start 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86f8db700 1 -- start start 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86f8db700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa868106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86f8db700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa868106c70 con 0x7fa8680601e0 2024-02-10T09:28:28.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86d677700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa868106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86d677700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa868106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:51076/0 (socket says 172.21.15.158:51076) 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.809+0000 7fa86d677700 1 -- 172.21.15.158:0/112630964 learned_addr learned my addr 172.21.15.158:0/112630964 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa86d677700 1 -- 172.21.15.158:0/112630964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa864005040 con 0x7fa8680601e0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa86d677700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa868106780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa864007a10 tx=0x7fa864009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa864018070 con 0x7fa8680601e0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa864013410 con 0x7fa8680601e0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa868106e70 con 0x7fa8680601e0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86401c3c0 con 0x7fa8680601e0 2024-02-10T09:28:28.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.810+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa868107300 con 0x7fa8680601e0 2024-02-10T09:28:28.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.811+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa864013ca0 con 0x7fa8680601e0 2024-02-10T09:28:28.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.811+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa868045ad0 con 0x7fa8680601e0 2024-02-10T09:28:28.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.811+0000 7fa85dffb700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa85402ff60 0x7fa854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:28.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.811+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8640414b0 con 0x7fa8680601e0 2024-02-10T09:28:28.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.811+0000 7fa86ce76700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa85402ff60 0x7fa854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:28.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.812+0000 7fa86ce76700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa85402ff60 0x7fa854032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fa8580096a0 tx=0x7fa858006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:28.817 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:28.816+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa86401d420 con 0x7fa8680601e0 2024-02-10T09:28:29.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.024+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa868108130 con 0x7fa8680601e0 2024-02-10T09:28:29.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.025+0000 7fa85dffb700 1 -- 172.21.15.158:0/112630964 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa864025ce0 con 0x7fa8680601e0 2024-02-10T09:28:29.026 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:29.026 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa85402ff60 msgr2=0x7fa854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa85402ff60 0x7fa854032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fa8580096a0 tx=0x7fa858006b40).stop 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 msgr2=0x7fa868106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa868106780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fa864007a10 tx=0x7fa864009b30).stop 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 shutdown_connections 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa85402ff60 0x7fa854032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 --2- 172.21.15.158:0/112630964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa8680601e0 0x7fa868106780 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 >> 172.21.15.158:0/112630964 conn(0x7fa8680f0b70 msgr2=0x7fa8680f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 shutdown_connections 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:29.027+0000 7fa86f8db700 1 -- 172.21.15.158:0/112630964 wait complete. 2024-02-10T09:28:29.029 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:30.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:28.091781+0000 mgr.smithi083.dowdym (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:29.026275+0000 mon.smithi083 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.158:0/112630964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:30.771 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:30.771 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:32.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:30.092162+0000 mgr.smithi083.dowdym (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 -- 172.21.15.158:0/2557760777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 msgr2=0x7fa7e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:32.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 --2- 172.21.15.158:0/2557760777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e80f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fa7dc009fb0 tx=0x7fa7dc009b30).stop 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 -- 172.21.15.158:0/2557760777 shutdown_connections 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 --2- 172.21.15.158:0/2557760777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e80f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 -- 172.21.15.158:0/2557760777 >> 172.21.15.158:0/2557760777 conn(0x7fa7e80f0b70 msgr2=0x7fa7e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 -- 172.21.15.158:0/2557760777 shutdown_connections 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 -- 172.21.15.158:0/2557760777 wait complete. 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.817+0000 7fa7ee07f700 1 Processor -- start 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7ee07f700 1 -- start start 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7ee07f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7ee07f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e8106cd0 con 0x7fa7e80f5410 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7e77fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7e77fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:50794/0 (socket says 172.21.15.158:50794) 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7e77fe700 1 -- 172.21.15.158:0/3680070318 learned_addr learned my addr 172.21.15.158:0/3680070318 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7e77fe700 1 -- 172.21.15.158:0/3680070318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc005040 con 0x7fa7e80f5410 2024-02-10T09:28:32.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.818+0000 7fa7e77fe700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e81067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa7dc0089f0 tx=0x7fa7dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.819+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7dc009320 con 0x7fa7e80f5410 2024-02-10T09:28:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.819+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e8106ed0 con 0x7fa7e80f5410 2024-02-10T09:28:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.819+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7dc013380 con 0x7fa7e80f5410 2024-02-10T09:28:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.819+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7dc01ed80 con 0x7fa7e80f5410 2024-02-10T09:28:32.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.819+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e8107360 con 0x7fa7e80f5410 2024-02-10T09:28:32.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.820+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7dc01e420 con 0x7fa7e80f5410 2024-02-10T09:28:32.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.820+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e8045ad0 con 0x7fa7e80f5410 2024-02-10T09:28:32.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.820+0000 7fa7cffff700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa7d002ff60 0x7fa7d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:32.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.820+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7dc040ed0 con 0x7fa7e80f5410 2024-02-10T09:28:32.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.820+0000 7fa7e6ffd700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa7d002ff60 0x7fa7d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:32.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.820+0000 7fa7e6ffd700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa7d002ff60 0x7fa7d0032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa7d80096a0 tx=0x7fa7d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:32.825 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:32.824+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7dc01c930 con 0x7fa7e80f5410 2024-02-10T09:28:33.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.037+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7e8108210 con 0x7fa7e80f5410 2024-02-10T09:28:33.039 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.038+0000 7fa7cffff700 1 -- 172.21.15.158:0/3680070318 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7dc01cb40 con 0x7fa7e80f5410 2024-02-10T09:28:33.040 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:33.040 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa7d002ff60 msgr2=0x7fa7d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa7d002ff60 0x7fa7d0032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa7d80096a0 tx=0x7fa7d8006b40).stop 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 msgr2=0x7fa7e81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e81067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa7dc0089f0 tx=0x7fa7dc00b040).stop 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 shutdown_connections 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa7d002ff60 0x7fa7d0032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 --2- 172.21.15.158:0/3680070318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7e80f5410 0x7fa7e81067e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 >> 172.21.15.158:0/3680070318 conn(0x7fa7e80f0b70 msgr2=0x7fa7e80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 shutdown_connections 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:33.040+0000 7fa7ee07f700 1 -- 172.21.15.158:0/3680070318 wait complete. 2024-02-10T09:28:33.042 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:34.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:32.092542+0000 mgr.smithi083.dowdym (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:34.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:33.039500+0000 mon.smithi083 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.158:0/3680070318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:34.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:34.747 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:36.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:34.092914+0000 mgr.smithi083.dowdym (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:36.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 -- 172.21.15.158:0/3168783005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 msgr2=0x7fce1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:36.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 --2- 172.21.15.158:0/3168783005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c0f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fce0c009fb0 tx=0x7fce0c009b30).stop 2024-02-10T09:28:36.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 -- 172.21.15.158:0/3168783005 shutdown_connections 2024-02-10T09:28:36.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 --2- 172.21.15.158:0/3168783005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c0f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:36.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 -- 172.21.15.158:0/3168783005 >> 172.21.15.158:0/3168783005 conn(0x7fce1c0f1c70 msgr2=0x7fce1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:36.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 -- 172.21.15.158:0/3168783005 shutdown_connections 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.667+0000 7fce23597700 1 -- 172.21.15.158:0/3168783005 wait complete. 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce23597700 1 Processor -- start 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce23597700 1 -- start start 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce23597700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce23597700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c106cd0 con 0x7fce1c0f6510 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce21333700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce21333700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:50810/0 (socket says 172.21.15.158:50810) 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce21333700 1 -- 172.21.15.158:0/2940947578 learned_addr learned my addr 172.21.15.158:0/2940947578 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.668+0000 7fce21333700 1 -- 172.21.15.158:0/2940947578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c005040 con 0x7fce1c0f6510 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.669+0000 7fce21333700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c1067e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fce0c0089f0 tx=0x7fce0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.669+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce0c009320 con 0x7fce1c0f6510 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.669+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c106ed0 con 0x7fce1c0f6510 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.669+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce0c013380 con 0x7fce1c0f6510 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.669+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce0c01ed80 con 0x7fce1c0f6510 2024-02-10T09:28:36.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.669+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c107360 con 0x7fce1c0f6510 2024-02-10T09:28:36.672 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.670+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce0c01e420 con 0x7fce1c0f6510 2024-02-10T09:28:36.672 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.670+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c045ad0 con 0x7fce1c0f6510 2024-02-10T09:28:36.672 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.670+0000 7fce11ffb700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce0802ff60 0x7fce08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:36.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.670+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce0c040f00 con 0x7fce1c0f6510 2024-02-10T09:28:36.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.670+0000 7fce20b32700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce0802ff60 0x7fce08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:36.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.671+0000 7fce20b32700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce0802ff60 0x7fce08032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fce180096a0 tx=0x7fce18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:36.676 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.675+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce0c0095f0 con 0x7fce1c0f6510 2024-02-10T09:28:36.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.881+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce1c108210 con 0x7fce1c0f6510 2024-02-10T09:28:36.882 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.881+0000 7fce11ffb700 1 -- 172.21.15.158:0/2940947578 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce0c01cb10 con 0x7fce1c0f6510 2024-02-10T09:28:36.883 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:36.883 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:36.884 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce0802ff60 msgr2=0x7fce08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce0802ff60 0x7fce08032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fce180096a0 tx=0x7fce18006b40).stop 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 msgr2=0x7fce1c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c1067e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fce0c0089f0 tx=0x7fce0c00b040).stop 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 shutdown_connections 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fce0802ff60 0x7fce08032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 --2- 172.21.15.158:0/2940947578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce1c0f6510 0x7fce1c1067e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 >> 172.21.15.158:0/2940947578 conn(0x7fce1c0f1c70 msgr2=0x7fce1c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 shutdown_connections 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:36.883+0000 7fce23597700 1 -- 172.21.15.158:0/2940947578 wait complete. 2024-02-10T09:28:36.885 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:38.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:36.093309+0000 mgr.smithi083.dowdym (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:38.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:36.882744+0000 mon.smithi083 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.158:0/2940947578' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:38.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:38.550 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:40.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:38.093614+0000 mgr.smithi083.dowdym (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 -- 172.21.15.158:0/2944107684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 msgr2=0x7f62cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 --2- 172.21.15.158:0/2944107684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc0f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f62c0009fb0 tx=0x7f62c0009b30).stop 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 -- 172.21.15.158:0/2944107684 shutdown_connections 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 --2- 172.21.15.158:0/2944107684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc0f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 -- 172.21.15.158:0/2944107684 >> 172.21.15.158:0/2944107684 conn(0x7f62cc0f0b70 msgr2=0x7f62cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 -- 172.21.15.158:0/2944107684 shutdown_connections 2024-02-10T09:28:40.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.480+0000 7f62d187e700 1 -- 172.21.15.158:0/2944107684 wait complete. 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.481+0000 7f62d187e700 1 Processor -- start 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.481+0000 7f62d187e700 1 -- start start 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.481+0000 7f62d187e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.481+0000 7f62d187e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc105c10 con 0x7f62cc0f5410 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62caffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62caffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52856/0 (socket says 172.21.15.158:52856) 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62caffd700 1 -- 172.21.15.158:0/4034857859 learned_addr learned my addr 172.21.15.158:0/4034857859 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62caffd700 1 -- 172.21.15.158:0/4034857859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c0005040 con 0x7f62cc0f5410 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62caffd700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc105720 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f62c0005010 tx=0x7f62c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c0009320 con 0x7f62cc0f5410 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc105e10 con 0x7f62cc0f5410 2024-02-10T09:28:40.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62c0013380 con 0x7f62cc0f5410 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62c0014e50 con 0x7f62cc0f5410 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.482+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc1062a0 con 0x7f62cc0f5410 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.483+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62c0014420 con 0x7f62cc0f5410 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.483+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc045ad0 con 0x7f62cc0f5410 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.483+0000 7f62b37fe700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f62b402ff10 0x7f62b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.483+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62c0040e30 con 0x7f62cc0f5410 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.484+0000 7f62ca7fc700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f62b402ff10 0x7f62b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:40.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.484+0000 7f62ca7fc700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f62b402ff10 0x7f62b40323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f62bc0096a0 tx=0x7f62bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:40.489 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.488+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62c00095f0 con 0x7f62cc0f5410 2024-02-10T09:28:40.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.693+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62cc107150 con 0x7f62cc0f5410 2024-02-10T09:28:40.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.694+0000 7f62b37fe700 1 -- 172.21.15.158:0/4034857859 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62c001db70 con 0x7f62cc0f5410 2024-02-10T09:28:40.695 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:40.695 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f62b402ff10 msgr2=0x7f62b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f62b402ff10 0x7f62b40323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f62bc0096a0 tx=0x7f62bc006b40).stop 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 msgr2=0x7f62cc105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc105720 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f62c0005010 tx=0x7f62c000b040).stop 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 shutdown_connections 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f62b402ff10 0x7f62b40323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 --2- 172.21.15.158:0/4034857859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62cc0f5410 0x7f62cc105720 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 >> 172.21.15.158:0/4034857859 conn(0x7f62cc0f0b70 msgr2=0x7f62cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 shutdown_connections 2024-02-10T09:28:40.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:40.695+0000 7f62d187e700 1 -- 172.21.15.158:0/4034857859 wait complete. 2024-02-10T09:28:40.698 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:41.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:40 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:40.695174+0000 mon.smithi083 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.158:0/4034857859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:42.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:40.093940+0000 mgr.smithi083.dowdym (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:42.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:42.485 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:44.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:42.094258+0000 mgr.smithi083.dowdym (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 -- 172.21.15.158:0/2398274753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 msgr2=0x7fc2f40f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 --2- 172.21.15.158:0/2398274753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f40f6db0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fc2e8009fb0 tx=0x7fc2e8009b30).stop 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 -- 172.21.15.158:0/2398274753 shutdown_connections 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 --2- 172.21.15.158:0/2398274753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f40f6db0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 -- 172.21.15.158:0/2398274753 >> 172.21.15.158:0/2398274753 conn(0x7fc2f40f0260 msgr2=0x7fc2f40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 -- 172.21.15.158:0/2398274753 shutdown_connections 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 -- 172.21.15.158:0/2398274753 wait complete. 2024-02-10T09:28:44.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.285+0000 7fc2f89af700 1 Processor -- start 2024-02-10T09:28:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f89af700 1 -- start start 2024-02-10T09:28:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f89af700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f89af700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f4106cb0 con 0x7fc2f40f6990 2024-02-10T09:28:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52874/0 (socket says 172.21.15.158:52874) 2024-02-10T09:28:44.288 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f259c700 1 -- 172.21.15.158:0/4121336371 learned_addr learned my addr 172.21.15.158:0/4121336371 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:44.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f259c700 1 -- 172.21.15.158:0/4121336371 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8005040 con 0x7fc2f40f6990 2024-02-10T09:28:44.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f259c700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f41067c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc2e8005010 tx=0x7fc2e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:44.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e8009320 con 0x7fc2f40f6990 2024-02-10T09:28:44.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f4106eb0 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2e8013380 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.286+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e8014e50 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.287+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f4107340 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.287+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2e80095f0 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.287+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f4045ad0 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.288+0000 7fc2e2ffd700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc2dc02ff10 0x7fc2dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.288+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2e8040fa0 con 0x7fc2f40f6990 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.288+0000 7fc2f1d9b700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc2dc02ff10 0x7fc2dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:44.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.288+0000 7fc2f1d9b700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc2dc02ff10 0x7fc2dc0323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc2e40096a0 tx=0x7fc2e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:44.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.292+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2e8014420 con 0x7fc2f40f6990 2024-02-10T09:28:44.497 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.496+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2f41081f0 con 0x7fc2f40f6990 2024-02-10T09:28:44.498 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.497+0000 7fc2e2ffd700 1 -- 172.21.15.158:0/4121336371 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2e8010230 con 0x7fc2f40f6990 2024-02-10T09:28:44.498 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:44.498 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc2dc02ff10 msgr2=0x7fc2dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc2dc02ff10 0x7fc2dc0323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc2e40096a0 tx=0x7fc2e4006b40).stop 2024-02-10T09:28:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 msgr2=0x7fc2f41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f41067c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc2e8005010 tx=0x7fc2e800b040).stop 2024-02-10T09:28:44.500 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 shutdown_connections 2024-02-10T09:28:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc2dc02ff10 0x7fc2dc0323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 --2- 172.21.15.158:0/4121336371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2f40f6990 0x7fc2f41067c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 >> 172.21.15.158:0/4121336371 conn(0x7fc2f40f0260 msgr2=0x7fc2f40f4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 shutdown_connections 2024-02-10T09:28:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:44.498+0000 7fc2f89af700 1 -- 172.21.15.158:0/4121336371 wait complete. 2024-02-10T09:28:44.501 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:45.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:44.498234+0000 mon.smithi083 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.158:0/4121336371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:46.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:46.105 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:44.094693+0000 mgr.smithi083.dowdym (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:48.153 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.150+0000 7f018267f700 1 -- 172.21.15.158:0/1007745422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 msgr2=0x7f017c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.150+0000 7f018267f700 1 --2- 172.21.15.158:0/1007745422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c0f90f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f016c009fb0 tx=0x7f016c009b30).stop 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.150+0000 7f018267f700 1 -- 172.21.15.158:0/1007745422 shutdown_connections 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.150+0000 7f018267f700 1 --2- 172.21.15.158:0/1007745422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c0f90f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.150+0000 7f018267f700 1 -- 172.21.15.158:0/1007745422 >> 172.21.15.158:0/1007745422 conn(0x7f017c0f0b70 msgr2=0x7f017c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.151+0000 7f018267f700 1 -- 172.21.15.158:0/1007745422 shutdown_connections 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.151+0000 7f018267f700 1 -- 172.21.15.158:0/1007745422 wait complete. 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.151+0000 7f018267f700 1 Processor -- start 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.151+0000 7f018267f700 1 -- start start 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.151+0000 7f018267f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.151+0000 7f018267f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c106c70 con 0x7f017c060280 2024-02-10T09:28:48.154 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f017bfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f017bfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52882/0 (socket says 172.21.15.158:52882) 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f017bfff700 1 -- 172.21.15.158:0/3752186069 learned_addr learned my addr 172.21.15.158:0/3752186069 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f017bfff700 1 -- 172.21.15.158:0/3752186069 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c005040 con 0x7f017c060280 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f017bfff700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c106780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f016c0089f0 tx=0x7f016c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016c009320 con 0x7f017c060280 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f016c013380 con 0x7f017c060280 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c106e70 con 0x7f017c060280 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f016c014e20 con 0x7f017c060280 2024-02-10T09:28:48.155 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.152+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c107300 con 0x7f017c060280 2024-02-10T09:28:48.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.153+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f016c0095f0 con 0x7f017c060280 2024-02-10T09:28:48.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.153+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c045ad0 con 0x7f017c060280 2024-02-10T09:28:48.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.153+0000 7f0178ff9700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f016402ff60 0x7f0164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:48.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.153+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f016c040f00 con 0x7f017c060280 2024-02-10T09:28:48.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.153+0000 7f017b7fe700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f016402ff60 0x7f0164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:48.156 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.154+0000 7f017b7fe700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f016402ff60 0x7f0164032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f01700096a0 tx=0x7f0170006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:48.159 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.158+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f016c014420 con 0x7f017c060280 2024-02-10T09:28:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:46.095136+0000 mgr.smithi083.dowdym (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.365+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f017c1081b0 con 0x7f017c060280 2024-02-10T09:28:48.367 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.366+0000 7f0178ff9700 1 -- 172.21.15.158:0/3752186069 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f016c010230 con 0x7f017c060280 2024-02-10T09:28:48.368 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:48.368 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:48.369 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f016402ff60 msgr2=0x7f0164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f016402ff60 0x7f0164032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f01700096a0 tx=0x7f0170006b40).stop 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 msgr2=0x7f017c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c106780 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f016c0089f0 tx=0x7f016c00b040).stop 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 shutdown_connections 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f016402ff60 0x7f0164032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 --2- 172.21.15.158:0/3752186069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f017c060280 0x7f017c106780 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 >> 172.21.15.158:0/3752186069 conn(0x7f017c0f0b70 msgr2=0x7f017c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 shutdown_connections 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:48.368+0000 7f018267f700 1 -- 172.21.15.158:0/3752186069 wait complete. 2024-02-10T09:28:48.370 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:49.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:48.367534+0000 mon.smithi083 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.158:0/3752186069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:49.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:49.960 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:50.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:48.095466+0000 mgr.smithi083.dowdym (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.825+0000 7fcba91cd700 1 -- 172.21.15.158:0/1481330188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 msgr2=0x7fcba40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.825+0000 7fcba91cd700 1 --2- 172.21.15.158:0/1481330188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba40f55c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fcb94009fb0 tx=0x7fcb94009b30).stop 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 -- 172.21.15.158:0/1481330188 shutdown_connections 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 --2- 172.21.15.158:0/1481330188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba40f55c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 -- 172.21.15.158:0/1481330188 >> 172.21.15.158:0/1481330188 conn(0x7fcba40f09a0 msgr2=0x7fcba40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 -- 172.21.15.158:0/1481330188 shutdown_connections 2024-02-10T09:28:51.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 -- 172.21.15.158:0/1481330188 wait complete. 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 Processor -- start 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 -- start start 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.826+0000 7fcba91cd700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba4106ac0 con 0x7fcba40f51a0 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba3fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba3fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47506/0 (socket says 172.21.15.158:47506) 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba3fff700 1 -- 172.21.15.158:0/912500006 learned_addr learned my addr 172.21.15.158:0/912500006 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:51.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba3fff700 1 -- 172.21.15.158:0/912500006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb94005040 con 0x7fcba40f51a0 2024-02-10T09:28:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba3fff700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba41065d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fcb940089f0 tx=0x7fcb9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb94009320 con 0x7fcba40f51a0 2024-02-10T09:28:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba4106cc0 con 0x7fcba40f51a0 2024-02-10T09:28:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb94013380 con 0x7fcba40f51a0 2024-02-10T09:28:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb94014e20 con 0x7fcba40f51a0 2024-02-10T09:28:51.830 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.827+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba4107150 con 0x7fcba40f51a0 2024-02-10T09:28:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.828+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb940263a0 con 0x7fcba40f51a0 2024-02-10T09:28:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.828+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba4045ad0 con 0x7fcba40f51a0 2024-02-10T09:28:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.828+0000 7fcba0ff9700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcb8c02ff10 0x7fcb8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.828+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb94041170 con 0x7fcba40f51a0 2024-02-10T09:28:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.829+0000 7fcba37fe700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcb8c02ff10 0x7fcb8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:51.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.829+0000 7fcba37fe700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcb8c02ff10 0x7fcb8c0323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcb980096a0 tx=0x7fcb98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:51.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:51.834+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb9401ccb0 con 0x7fcba40f51a0 2024-02-10T09:28:52.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.036+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcba40fad30 con 0x7fcba40f51a0 2024-02-10T09:28:52.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.036+0000 7fcba0ff9700 1 -- 172.21.15.158:0/912500006 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb9401cad0 con 0x7fcba40f51a0 2024-02-10T09:28:52.078 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:52.078 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcb8c02ff10 msgr2=0x7fcb8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcb8c02ff10 0x7fcb8c0323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcb980096a0 tx=0x7fcb98006b40).stop 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 msgr2=0x7fcba41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba41065d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fcb940089f0 tx=0x7fcb9400b040).stop 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 shutdown_connections 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcb8c02ff10 0x7fcb8c0323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 --2- 172.21.15.158:0/912500006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcba40f51a0 0x7fcba41065d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:52.079 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 >> 172.21.15.158:0/912500006 conn(0x7fcba40f09a0 msgr2=0x7fcba405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:52.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 shutdown_connections 2024-02-10T09:28:52.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:52.038+0000 7fcba91cd700 1 -- 172.21.15.158:0/912500006 wait complete. 2024-02-10T09:28:52.080 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:52.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:50.095755+0000 mgr.smithi083.dowdym (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:53.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:52 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:52.037967+0000 mon.smithi083 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.158:0/912500006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:53.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:53.530 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:54.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:52.096148+0000 mgr.smithi083.dowdym (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.337+0000 7f978d606700 1 -- 172.21.15.158:0/1273336267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 msgr2=0x7f97880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.337+0000 7f978d606700 1 --2- 172.21.15.158:0/1273336267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f97880f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f9778009fb0 tx=0x7f9778009b30).stop 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.337+0000 7f978d606700 1 -- 172.21.15.158:0/1273336267 shutdown_connections 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.337+0000 7f978d606700 1 --2- 172.21.15.158:0/1273336267 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f97880f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 -- 172.21.15.158:0/1273336267 >> 172.21.15.158:0/1273336267 conn(0x7f97880f0b70 msgr2=0x7f97880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 -- 172.21.15.158:0/1273336267 shutdown_connections 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 -- 172.21.15.158:0/1273336267 wait complete. 2024-02-10T09:28:55.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 Processor -- start 2024-02-10T09:28:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 -- start start 2024-02-10T09:28:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f9788106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.338+0000 7f978d606700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9788106d00 con 0x7f97880f5410 2024-02-10T09:28:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f9786ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f9788106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f9786ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f9788106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47524/0 (socket says 172.21.15.158:47524) 2024-02-10T09:28:55.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f9786ffd700 1 -- 172.21.15.158:0/2051010155 learned_addr learned my addr 172.21.15.158:0/2051010155 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:55.342 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f9786ffd700 1 -- 172.21.15.158:0/2051010155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9778005040 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f9786ffd700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f9788106810 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f9778005010 tx=0x7f977800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778009320 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9778013380 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9788106f00 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9778014e50 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.339+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9788107390 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.340+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9778014420 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.340+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9788045ad0 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.340+0000 7f976f7fe700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f977002ff10 0x7f97700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.340+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9778040e30 con 0x7f97880f5410 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.340+0000 7f97867fc700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f977002ff10 0x7f97700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:55.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.341+0000 7f97867fc700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f977002ff10 0x7f97700323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f977c0096a0 tx=0x7f977c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:55.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.346+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97780095f0 con 0x7f97880f5410 2024-02-10T09:28:55.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.551+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97880fafa0 con 0x7f97880f5410 2024-02-10T09:28:55.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.551+0000 7f976f7fe700 1 -- 172.21.15.158:0/2051010155 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f977801db70 con 0x7f97880f5410 2024-02-10T09:28:55.553 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:55.553 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:55.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f977002ff10 msgr2=0x7f97700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f977002ff10 0x7f97700323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f977c0096a0 tx=0x7f977c006b40).stop 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 msgr2=0x7f9788106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f9788106810 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f9778005010 tx=0x7f977800b040).stop 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 shutdown_connections 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f977002ff10 0x7f97700323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 --2- 172.21.15.158:0/2051010155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f97880f5410 0x7f9788106810 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 >> 172.21.15.158:0/2051010155 conn(0x7f97880f0b70 msgr2=0x7f978805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 shutdown_connections 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:55.553+0000 7f978d606700 1 -- 172.21.15.158:0/2051010155 wait complete. 2024-02-10T09:28:55.555 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:28:56.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:54.096542+0000 mgr.smithi083.dowdym (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:56.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:55.552982+0000 mon.smithi083 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.158:0/2051010155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:28:57.219 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:28:57.220 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:28:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:56.096957+0000 mgr.smithi083.dowdym (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- 172.21.15.158:0/4073022344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 msgr2=0x7f0eac0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 --2- 172.21.15.158:0/4073022344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac0f66c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f0e9c009fb0 tx=0x7f0e9c009b30).stop 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- 172.21.15.158:0/4073022344 shutdown_connections 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 --2- 172.21.15.158:0/4073022344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac0f66c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- 172.21.15.158:0/4073022344 >> 172.21.15.158:0/4073022344 conn(0x7f0eac0f1aa0 msgr2=0x7f0eac0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- 172.21.15.158:0/4073022344 shutdown_connections 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- 172.21.15.158:0/4073022344 wait complete. 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 Processor -- start 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- start start 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:59.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.125+0000 7f0eb27b3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac069e20 con 0x7f0eac0f62a0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb17b1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb17b1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47542/0 (socket says 172.21.15.158:47542) 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb17b1700 1 -- 172.21.15.158:0/3223044140 learned_addr learned my addr 172.21.15.158:0/3223044140 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb17b1700 1 -- 172.21.15.158:0/3223044140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c005040 con 0x7f0eac0f62a0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb17b1700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f0e9c0089f0 tx=0x7f0e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e9c009320 con 0x7f0eac0f62a0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac069fc0 con 0x7f0eac0f62a0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e9c013380 con 0x7f0eac0f62a0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e9c014e20 con 0x7f0eac0f62a0 2024-02-10T09:28:59.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.126+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac06a450 con 0x7f0eac0f62a0 2024-02-10T09:28:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.127+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e9c0095f0 con 0x7f0eac0f62a0 2024-02-10T09:28:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.127+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac059100 con 0x7f0eac0f62a0 2024-02-10T09:28:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.127+0000 7f0ea1ffb700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0e9802ff10 0x7f0e980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:28:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.127+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e9c040e90 con 0x7f0eac0f62a0 2024-02-10T09:28:59.129 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.128+0000 7f0eb0fb0700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0e9802ff10 0x7f0e980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:28:59.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.128+0000 7f0eb0fb0700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0e9802ff10 0x7f0e980323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0ea80096a0 tx=0x7f0ea8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:28:59.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.132+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e9c014420 con 0x7f0eac0f62a0 2024-02-10T09:28:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.340+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0eac0fbe30 con 0x7f0eac0f62a0 2024-02-10T09:28:59.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.340+0000 7f0ea1ffb700 1 -- 172.21.15.158:0/3223044140 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e9c01d980 con 0x7f0eac0f62a0 2024-02-10T09:28:59.341 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:28:59.341 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:28:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0e9802ff10 msgr2=0x7f0e980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0e9802ff10 0x7f0e980323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f0ea80096a0 tx=0x7f0ea8006b40).stop 2024-02-10T09:28:59.342 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 msgr2=0x7f0eac06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f0e9c0089f0 tx=0x7f0e9c00b040).stop 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 shutdown_connections 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0e9802ff10 0x7f0e980323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 --2- 172.21.15.158:0/3223044140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0eac0f62a0 0x7f0eac06b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 >> 172.21.15.158:0/3223044140 conn(0x7f0eac0f1aa0 msgr2=0x7f0eac0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 shutdown_connections 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:28:59.342+0000 7f0eb27b3700 1 -- 172.21.15.158:0/3223044140 wait complete. 2024-02-10T09:28:59.343 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:28:58.097268+0000 mgr.smithi083.dowdym (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:28:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:28:59.340674+0000 mon.smithi083 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.158:0/3223044140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:01.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:01.025 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:02.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:00.097690+0000 mgr.smithi083.dowdym (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:03.172 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 -- 172.21.15.158:0/988265745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 msgr2=0x7fa7ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:03.172 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 --2- 172.21.15.158:0/988265745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fa7a8009fb0 tx=0x7fa7a8009b30).stop 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 -- 172.21.15.158:0/988265745 shutdown_connections 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 --2- 172.21.15.158:0/988265745 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 -- 172.21.15.158:0/988265745 >> 172.21.15.158:0/988265745 conn(0x7fa7ac0f0b70 msgr2=0x7fa7ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 -- 172.21.15.158:0/988265745 shutdown_connections 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.158+0000 7fa7b45e8700 1 -- 172.21.15.158:0/988265745 wait complete. 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b45e8700 1 Processor -- start 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b45e8700 1 -- start start 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b45e8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b45e8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac106cd0 con 0x7fa7ac0f5410 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b2384700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b2384700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53444/0 (socket says 172.21.15.158:53444) 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b2384700 1 -- 172.21.15.158:0/2864717338 learned_addr learned my addr 172.21.15.158:0/2864717338 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:03.173 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b2384700 1 -- 172.21.15.158:0/2864717338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8005040 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.159+0000 7fa7b2384700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac1067e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fa7a80089f0 tx=0x7fa7a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a8009320 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ac106ed0 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa7a8013380 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a801ed80 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ac107360 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7a801e420 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.160+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ac045ad0 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.161+0000 7fa7a2ffd700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa79802ff60 0x7fa798032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.161+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7a8041050 con 0x7fa7ac0f5410 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.161+0000 7fa7b1b83700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa79802ff60 0x7fa798032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.161+0000 7fa7b1b83700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa79802ff60 0x7fa798032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa79c0096a0 tx=0x7fa79c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:03.174 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.165+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7a801cb10 con 0x7fa7ac0f5410 2024-02-10T09:29:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.373+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7ac108210 con 0x7fa7ac0f5410 2024-02-10T09:29:03.374 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.374+0000 7fa7a2ffd700 1 -- 172.21.15.158:0/2864717338 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa7a8004490 con 0x7fa7ac0f5410 2024-02-10T09:29:03.375 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:03.375 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa79802ff60 msgr2=0x7fa798032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa79802ff60 0x7fa798032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa79c0096a0 tx=0x7fa79c006b40).stop 2024-02-10T09:29:03.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 msgr2=0x7fa7ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac1067e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fa7a80089f0 tx=0x7fa7a800b040).stop 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 shutdown_connections 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fa79802ff60 0x7fa798032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 --2- 172.21.15.158:0/2864717338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa7ac0f5410 0x7fa7ac1067e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 >> 172.21.15.158:0/2864717338 conn(0x7fa7ac0f0b70 msgr2=0x7fa7ac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 shutdown_connections 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:03.376+0000 7fa7b45e8700 1 -- 172.21.15.158:0/2864717338 wait complete. 2024-02-10T09:29:03.377 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:04.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:02.097953+0000 mgr.smithi083.dowdym (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:04.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:03.374699+0000 mon.smithi083 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.158:0/2864717338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:05.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:05.082 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:06.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:04.098213+0000 mgr.smithi083.dowdym (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/3148241729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 msgr2=0x7fcc180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 --2- 172.21.15.158:0/3148241729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc180f5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fcc14009fb0 tx=0x7fcc14009b30).stop 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/3148241729 shutdown_connections 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 --2- 172.21.15.158:0/3148241729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc180f5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/3148241729 >> 172.21.15.158:0/3148241729 conn(0x7fcc180f0b70 msgr2=0x7fcc180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/3148241729 shutdown_connections 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.020+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/3148241729 wait complete. 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1f9ed700 1 Processor -- start 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1f9ed700 1 -- start start 2024-02-10T09:29:07.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1f9ed700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc181094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:07.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1f9ed700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc18060240 con 0x7fcc180f5410 2024-02-10T09:29:07.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1d789700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc181094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:07.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1d789700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc181094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53456/0 (socket says 172.21.15.158:53456) 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1d789700 1 -- 172.21.15.158:0/1502500321 learned_addr learned my addr 172.21.15.158:0/1502500321 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1d789700 1 -- 172.21.15.158:0/1502500321 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc14005040 con 0x7fcc180f5410 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.021+0000 7fcc1d789700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc181094e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcc14005010 tx=0x7fcc1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.022+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc14009320 con 0x7fcc180f5410 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.022+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc18060440 con 0x7fcc180f5410 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.022+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc14013380 con 0x7fcc180f5410 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.022+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc14014e50 con 0x7fcc180f5410 2024-02-10T09:29:07.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.022+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc18109b90 con 0x7fcc180f5410 2024-02-10T09:29:07.024 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.023+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18045ad0 con 0x7fcc180f5410 2024-02-10T09:29:07.024 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.023+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc140095f0 con 0x7fcc180f5410 2024-02-10T09:29:07.024 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.023+0000 7fcc0dffb700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc0402ff10 0x7fcc040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:07.024 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.023+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc14040e70 con 0x7fcc180f5410 2024-02-10T09:29:07.024 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.023+0000 7fcc1cf88700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc0402ff10 0x7fcc040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:07.024 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.023+0000 7fcc1cf88700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc0402ff10 0x7fcc040323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcc080096a0 tx=0x7fcc08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:07.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.027+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc14014420 con 0x7fcc180f5410 2024-02-10T09:29:07.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.233+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc1810a3a0 con 0x7fcc180f5410 2024-02-10T09:29:07.234 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.234+0000 7fcc0dffb700 1 -- 172.21.15.158:0/1502500321 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc14010230 con 0x7fcc180f5410 2024-02-10T09:29:07.235 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:07.235 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:07.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc0402ff10 msgr2=0x7fcc040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:07.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc0402ff10 0x7fcc040323d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fcc080096a0 tx=0x7fcc08006b40).stop 2024-02-10T09:29:07.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 msgr2=0x7fcc181094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:07.236 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc181094e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcc14005010 tx=0x7fcc1400b040).stop 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 shutdown_connections 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc0402ff10 0x7fcc040323d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 --2- 172.21.15.158:0/1502500321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc180f5410 0x7fcc181094e0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 >> 172.21.15.158:0/1502500321 conn(0x7fcc180f0b70 msgr2=0x7fcc180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 shutdown_connections 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:07.236+0000 7fcc1f9ed700 1 -- 172.21.15.158:0/1502500321 wait complete. 2024-02-10T09:29:07.237 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:08.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:06.098655+0000 mgr.smithi083.dowdym (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:08.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:07.234646+0000 mon.smithi083 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.158:0/1502500321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:08.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:08.947 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:10.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:08.099043+0000 mgr.smithi083.dowdym (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.774+0000 7f9ce293e700 1 -- 172.21.15.158:0/3586225629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 msgr2=0x7f9cdc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.774+0000 7f9ce293e700 1 --2- 172.21.15.158:0/3586225629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f55c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f9cd8009fb0 tx=0x7f9cd8009b30).stop 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.774+0000 7f9ce293e700 1 -- 172.21.15.158:0/3586225629 shutdown_connections 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.774+0000 7f9ce293e700 1 --2- 172.21.15.158:0/3586225629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f55c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.774+0000 7f9ce293e700 1 -- 172.21.15.158:0/3586225629 >> 172.21.15.158:0/3586225629 conn(0x7f9cdc0f09a0 msgr2=0x7f9cdc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.775+0000 7f9ce293e700 1 -- 172.21.15.158:0/3586225629 shutdown_connections 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.775+0000 7f9ce293e700 1 -- 172.21.15.158:0/3586225629 wait complete. 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.775+0000 7f9ce293e700 1 Processor -- start 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.775+0000 7f9ce293e700 1 -- start start 2024-02-10T09:29:10.777 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.775+0000 7f9ce293e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.775+0000 7f9ce293e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0f13a0 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce193c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce193c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40784/0 (socket says 172.21.15.158:40784) 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce193c700 1 -- 172.21.15.158:0/3859815885 learned_addr learned my addr 172.21.15.158:0/3859815885 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce193c700 1 -- 172.21.15.158:0/3859815885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8005040 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce193c700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f2f20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f9cd80089f0 tx=0x7f9cd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd8009320 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc0f1540 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd8013380 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd8014e20 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.776+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc0f19d0 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.777+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cd80095f0 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.777+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc059100 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.777+0000 7f9cd27fc700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9cc802ff10 0x7f9cc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:10.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.777+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cd8040e90 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.777+0000 7f9ce113b700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9cc802ff10 0x7f9cc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:10.780 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.778+0000 7f9ce113b700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9cc802ff10 0x7f9cc80323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9ccc0096a0 tx=0x7f9ccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:10.782 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.782+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cd8014420 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.984+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9cdc045ad0 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.984 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.984+0000 7f9cd27fc700 1 -- 172.21.15.158:0/3859815885 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9cd801d980 con 0x7f9cdc0f51a0 2024-02-10T09:29:10.985 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:10.985 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9cc802ff10 msgr2=0x7f9cc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9cc802ff10 0x7f9cc80323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9ccc0096a0 tx=0x7f9ccc006b40).stop 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 msgr2=0x7f9cdc0f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f2f20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f9cd80089f0 tx=0x7f9cd800b040).stop 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 shutdown_connections 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9cc802ff10 0x7f9cc80323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 --2- 172.21.15.158:0/3859815885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9cdc0f51a0 0x7f9cdc0f2f20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 >> 172.21.15.158:0/3859815885 conn(0x7f9cdc0f09a0 msgr2=0x7f9cdc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 shutdown_connections 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:10.986+0000 7f9ce293e700 1 -- 172.21.15.158:0/3859815885 wait complete. 2024-02-10T09:29:10.987 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:12.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:10.099454+0000 mgr.smithi083.dowdym (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:12.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:10.984677+0000 mon.smithi083 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.158:0/3859815885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:12.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:12.779 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:14.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:12.099734+0000 mgr.smithi083.dowdym (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:14.625 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 -- 172.21.15.158:0/4181623232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 msgr2=0x7f94ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:14.625 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 --2- 172.21.15.158:0/4181623232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec0f55c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f94dc009fb0 tx=0x7f94dc009b30).stop 2024-02-10T09:29:14.625 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 -- 172.21.15.158:0/4181623232 shutdown_connections 2024-02-10T09:29:14.625 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 --2- 172.21.15.158:0/4181623232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec0f55c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:14.625 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 -- 172.21.15.158:0/4181623232 >> 172.21.15.158:0/4181623232 conn(0x7f94ec0f09a0 msgr2=0x7f94ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 -- 172.21.15.158:0/4181623232 shutdown_connections 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.620+0000 7f94f31c1700 1 -- 172.21.15.158:0/4181623232 wait complete. 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f31c1700 1 Processor -- start 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f31c1700 1 -- start start 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f31c1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec10b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f31c1700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec1070e0 con 0x7f94ec0f51a0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f21bf700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec10b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f21bf700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec10b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40798/0 (socket says 172.21.15.158:40798) 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f21bf700 1 -- 172.21.15.158:0/1475129230 learned_addr learned my addr 172.21.15.158:0/1475129230 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f21bf700 1 -- 172.21.15.158:0/1475129230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc005040 con 0x7f94ec0f51a0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.621+0000 7f94f21bf700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec10b6f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f94dc009830 tx=0x7f94dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.622+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94dc009320 con 0x7f94ec0f51a0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.622+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec060220 con 0x7f94ec0f51a0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.622+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94dc013380 con 0x7f94ec0f51a0 2024-02-10T09:29:14.626 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.622+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94dc014420 con 0x7f94ec0f51a0 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.622+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec10bbe0 con 0x7f94ec0f51a0 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.623+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94dc01e420 con 0x7f94ec0f51a0 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.622+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec045ad0 con 0x7f94ec0f51a0 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.623+0000 7f94e2ffd700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f94d802ff60 0x7f94d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.623+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94dc040e30 con 0x7f94ec0f51a0 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.623+0000 7f94f19be700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f94d802ff60 0x7f94d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:14.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.623+0000 7f94f19be700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f94d802ff60 0x7f94d8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f94e80096a0 tx=0x7f94e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:14.628 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.628+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94dc01cb10 con 0x7f94ec0f51a0 2024-02-10T09:29:14.832 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.832+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94ec0fad30 con 0x7f94ec0f51a0 2024-02-10T09:29:14.833 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.832+0000 7f94e2ffd700 1 -- 172.21.15.158:0/1475129230 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94dc004490 con 0x7f94ec0f51a0 2024-02-10T09:29:14.833 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:14.833 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f94d802ff60 msgr2=0x7f94d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f94d802ff60 0x7f94d8032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f94e80096a0 tx=0x7f94e8006b40).stop 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 msgr2=0x7f94ec10b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec10b6f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f94dc009830 tx=0x7f94dc00b040).stop 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 shutdown_connections 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f94d802ff60 0x7f94d8032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:14.834 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 --2- 172.21.15.158:0/1475129230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94ec0f51a0 0x7f94ec10b6f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 >> 172.21.15.158:0/1475129230 conn(0x7f94ec0f09a0 msgr2=0x7f94ec0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 shutdown_connections 2024-02-10T09:29:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:14.834+0000 7f94f31c1700 1 -- 172.21.15.158:0/1475129230 wait complete. 2024-02-10T09:29:14.835 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:14.172714+0000 mon.smithi083 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:29:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:14.180628+0000 mon.smithi083 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:29:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:14.100123+0000 mgr.smithi083.dowdym (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:14.832689+0000 mon.smithi083 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.158:0/1475129230' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:16.529 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:16.529 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:18.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:16.100435+0000 mgr.smithi083.dowdym (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 -- 172.21.15.158:0/979965350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 msgr2=0x7fb9fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 --2- 172.21.15.158:0/979965350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc0f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fb9ec009fb0 tx=0x7fb9ec009b30).stop 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 -- 172.21.15.158:0/979965350 shutdown_connections 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 --2- 172.21.15.158:0/979965350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc0f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 -- 172.21.15.158:0/979965350 >> 172.21.15.158:0/979965350 conn(0x7fb9fc0f0b70 msgr2=0x7fb9fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 -- 172.21.15.158:0/979965350 shutdown_connections 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 -- 172.21.15.158:0/979965350 wait complete. 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 Processor -- start 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.400+0000 7fba045dd700 1 -- start start 2024-02-10T09:29:18.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba045dd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba045dd700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9fc105bd0 con 0x7fb9fc0f5410 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba02379700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba02379700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40822/0 (socket says 172.21.15.158:40822) 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba02379700 1 -- 172.21.15.158:0/1107044715 learned_addr learned my addr 172.21.15.158:0/1107044715 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba02379700 1 -- 172.21.15.158:0/1107044715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec005040 con 0x7fb9fc0f5410 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba02379700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc1056e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fb9ec0089f0 tx=0x7fb9ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ec009320 con 0x7fb9fc0f5410 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9fc105dd0 con 0x7fb9fc0f5410 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9ec013380 con 0x7fb9fc0f5410 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9ec01ed80 con 0x7fb9fc0f5410 2024-02-10T09:29:18.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.401+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9fc106260 con 0x7fb9fc0f5410 2024-02-10T09:29:18.404 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.402+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9ec01e420 con 0x7fb9fc0f5410 2024-02-10T09:29:18.404 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.402+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9fc045ad0 con 0x7fb9fc0f5410 2024-02-10T09:29:18.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.402+0000 7fb9f2ffd700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9e802ff60 0x7fb9e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:18.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.403+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9ec040e70 con 0x7fb9fc0f5410 2024-02-10T09:29:18.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.403+0000 7fba01b78700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9e802ff60 0x7fb9e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:18.405 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.403+0000 7fba01b78700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9e802ff60 0x7fb9e8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb9f80096a0 tx=0x7fb9f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:18.407 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.407+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9ec00e2a0 con 0x7fb9fc0f5410 2024-02-10T09:29:18.614 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.614+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9fc107110 con 0x7fb9fc0f5410 2024-02-10T09:29:18.615 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.615+0000 7fb9f2ffd700 1 -- 172.21.15.158:0/1107044715 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9ec00e2a0 con 0x7fb9fc0f5410 2024-02-10T09:29:18.615 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:18.615 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9e802ff60 msgr2=0x7fb9e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9e802ff60 0x7fb9e8032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fb9f80096a0 tx=0x7fb9f8006b40).stop 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 msgr2=0x7fb9fc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc1056e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fb9ec0089f0 tx=0x7fb9ec00b040).stop 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 shutdown_connections 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb9e802ff60 0x7fb9e8032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 --2- 172.21.15.158:0/1107044715 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9fc0f5410 0x7fb9fc1056e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 >> 172.21.15.158:0/1107044715 conn(0x7fb9fc0f0b70 msgr2=0x7fb9fc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:18.617 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.616+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 shutdown_connections 2024-02-10T09:29:18.618 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:18.617+0000 7fba045dd700 1 -- 172.21.15.158:0/1107044715 wait complete. 2024-02-10T09:29:18.618 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:19.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:18 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:18.615065+0000 mon.smithi083 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.158:0/1107044715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:18.100805+0000 mgr.smithi083.dowdym (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:20.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:20.341 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:22.208 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.199+0000 7fd0ee65d700 1 -- 172.21.15.158:0/166465577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 msgr2=0x7fd0e80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.199+0000 7fd0ee65d700 1 --2- 172.21.15.158:0/166465577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e80f68a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fd0dc009fb0 tx=0x7fd0dc009b30).stop 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.199+0000 7fd0ee65d700 1 -- 172.21.15.158:0/166465577 shutdown_connections 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.199+0000 7fd0ee65d700 1 --2- 172.21.15.158:0/166465577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e80f68a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 -- 172.21.15.158:0/166465577 >> 172.21.15.158:0/166465577 conn(0x7fd0e80f1c00 msgr2=0x7fd0e80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 -- 172.21.15.158:0/166465577 shutdown_connections 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 -- 172.21.15.158:0/166465577 wait complete. 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 Processor -- start 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 -- start start 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.200+0000 7fd0ee65d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e8106ce0 con 0x7fd0e80f6480 2024-02-10T09:29:22.209 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47062/0 (socket says 172.21.15.158:47062) 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e7fff700 1 -- 172.21.15.158:0/403430136 learned_addr learned my addr 172.21.15.158:0/403430136 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e7fff700 1 -- 172.21.15.158:0/403430136 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc005040 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e7fff700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e81067f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd0dc0089f0 tx=0x7fd0dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0dc009320 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e8106ee0 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0dc013380 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0dc01ed80 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.201+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e8107370 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.202+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0dc01e420 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.202+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e8045ad0 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.203+0000 7fd0e4ff9700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd0d002ff60 0x7fd0d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.203+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0dc0410b0 con 0x7fd0e80f6480 2024-02-10T09:29:22.210 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.203+0000 7fd0e77fe700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd0d002ff60 0x7fd0d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:22.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.203+0000 7fd0e77fe700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd0d002ff60 0x7fd0d0032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd0d80096a0 tx=0x7fd0d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:22.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.207+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd0dc0095f0 con 0x7fd0e80f6480 2024-02-10T09:29:22.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:20.101128+0000 mgr.smithi083.dowdym (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:22.416 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.416+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0e80fc090 con 0x7fd0e80f6480 2024-02-10T09:29:22.416 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.416+0000 7fd0e4ff9700 1 -- 172.21.15.158:0/403430136 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd0dc01cb10 con 0x7fd0e80f6480 2024-02-10T09:29:22.416 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:22.417 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:22.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd0d002ff60 msgr2=0x7fd0d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd0d002ff60 0x7fd0d0032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fd0d80096a0 tx=0x7fd0d8006b40).stop 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 msgr2=0x7fd0e81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e81067f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fd0dc0089f0 tx=0x7fd0dc00b040).stop 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 shutdown_connections 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd0d002ff60 0x7fd0d0032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 --2- 172.21.15.158:0/403430136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0e80f6480 0x7fd0e81067f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 >> 172.21.15.158:0/403430136 conn(0x7fd0e80f1c00 msgr2=0x7fd0e80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 shutdown_connections 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:22.418+0000 7fd0ee65d700 1 -- 172.21.15.158:0/403430136 wait complete. 2024-02-10T09:29:22.419 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:23.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:22.416728+0000 mon.smithi083 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.158:0/403430136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:24.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:24.217 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:22.101479+0000 mgr.smithi083.dowdym (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 -- 172.21.15.158:0/4083849994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 msgr2=0x7fdbb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 --2- 172.21.15.158:0/4083849994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb80f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fdbb4009fb0 tx=0x7fdbb4009b30).stop 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 -- 172.21.15.158:0/4083849994 shutdown_connections 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 --2- 172.21.15.158:0/4083849994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb80f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 -- 172.21.15.158:0/4083849994 >> 172.21.15.158:0/4083849994 conn(0x7fdbb80f09a0 msgr2=0x7fdbb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 -- 172.21.15.158:0/4083849994 shutdown_connections 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 -- 172.21.15.158:0/4083849994 wait complete. 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 Processor -- start 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.199+0000 7fdbbef4e700 1 -- start start 2024-02-10T09:29:26.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbef4e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbef4e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8060220 con 0x7fdbb80f51a0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbdf4c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbdf4c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47078/0 (socket says 172.21.15.158:47078) 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbdf4c700 1 -- 172.21.15.158:0/3151887385 learned_addr learned my addr 172.21.15.158:0/3151887385 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbdf4c700 1 -- 172.21.15.158:0/3151887385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb4005040 con 0x7fdbb80f51a0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbdf4c700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb81092f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdbb40089f0 tx=0x7fdbb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb4009320 con 0x7fdbb80f51a0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb8060420 con 0x7fdbb80f51a0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbb4013380 con 0x7fdbb80f51a0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb4014e20 con 0x7fdbb80f51a0 2024-02-10T09:29:26.203 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.200+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb81099a0 con 0x7fdbb80f51a0 2024-02-10T09:29:26.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.201+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbb4014420 con 0x7fdbb80f51a0 2024-02-10T09:29:26.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.201+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb8059100 con 0x7fdbb80f51a0 2024-02-10T09:29:26.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.201+0000 7fdbae7fc700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdba402ff10 0x7fdba40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:26.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.201+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbb4040fc0 con 0x7fdbb80f51a0 2024-02-10T09:29:26.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.202+0000 7fdbbd74b700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdba402ff10 0x7fdba40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:26.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.202+0000 7fdbbd74b700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdba402ff10 0x7fdba40323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fdba80096a0 tx=0x7fdba8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:26.207 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.206+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbb406d2b0 con 0x7fdbb80f51a0 2024-02-10T09:29:26.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:24.101839+0000 mgr.smithi083.dowdym (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:26.410 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.409+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbb8045ad0 con 0x7fdbb80f51a0 2024-02-10T09:29:26.411 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.410+0000 7fdbae7fc700 1 -- 172.21.15.158:0/3151887385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbb40095f0 con 0x7fdbb80f51a0 2024-02-10T09:29:26.412 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:26.412 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdba402ff10 msgr2=0x7fdba40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdba402ff10 0x7fdba40323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fdba80096a0 tx=0x7fdba8006b40).stop 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 msgr2=0x7fdbb81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb81092f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fdbb40089f0 tx=0x7fdbb400b040).stop 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 shutdown_connections 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fdba402ff10 0x7fdba40323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 --2- 172.21.15.158:0/3151887385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdbb80f51a0 0x7fdbb81092f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:26.413 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 >> 172.21.15.158:0/3151887385 conn(0x7fdbb80f09a0 msgr2=0x7fdbb80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:26.414 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 shutdown_connections 2024-02-10T09:29:26.414 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:26.412+0000 7fdbbef4e700 1 -- 172.21.15.158:0/3151887385 wait complete. 2024-02-10T09:29:26.414 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:27.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:26.411299+0000 mon.smithi083 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.158:0/3151887385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:28.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:28.043 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:28.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:26.102201+0000 mgr.smithi083.dowdym (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 -- 172.21.15.158:0/3792038470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 msgr2=0x7fd1b00f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 --2- 172.21.15.158:0/3792038470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b00f5750 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fd1a4009fb0 tx=0x7fd1a4009b30).stop 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 -- 172.21.15.158:0/3792038470 shutdown_connections 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 --2- 172.21.15.158:0/3792038470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b00f5750 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 -- 172.21.15.158:0/3792038470 >> 172.21.15.158:0/3792038470 conn(0x7fd1b00f0a90 msgr2=0x7fd1b00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 -- 172.21.15.158:0/3792038470 shutdown_connections 2024-02-10T09:29:30.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.042+0000 7fd1b722b700 1 -- 172.21.15.158:0/3792038470 wait complete. 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.043+0000 7fd1b722b700 1 Processor -- start 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.043+0000 7fd1b722b700 1 -- start start 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.043+0000 7fd1b722b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b0109780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.043+0000 7fd1b722b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b0060240 con 0x7fd1b00f5330 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b4fc7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b0109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b4fc7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b0109780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56826/0 (socket says 172.21.15.158:56826) 2024-02-10T09:29:30.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b4fc7700 1 -- 172.21.15.158:0/2092135228 learned_addr learned my addr 172.21.15.158:0/2092135228 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b4fc7700 1 -- 172.21.15.158:0/2092135228 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a4005040 con 0x7fd1b00f5330 2024-02-10T09:29:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b4fc7700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b0109780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fd1a40089f0 tx=0x7fd1a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1a4009320 con 0x7fd1b00f5330 2024-02-10T09:29:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b0060440 con 0x7fd1b00f5330 2024-02-10T09:29:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1a4013380 con 0x7fd1b00f5330 2024-02-10T09:29:30.047 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1a401ed80 con 0x7fd1b00f5330 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.044+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b0109e30 con 0x7fd1b00f5330 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.045+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd1a4014420 con 0x7fd1b00f5330 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.045+0000 7fd1ad7fa700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd1a002ff60 0x7fd1a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.045+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b0045ad0 con 0x7fd1b00f5330 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.046+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd1a4041120 con 0x7fd1b00f5330 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.046+0000 7fd1affff700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd1a002ff60 0x7fd1a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:30.048 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.046+0000 7fd1affff700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd1a002ff60 0x7fd1a0032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fd19c0096a0 tx=0x7fd19c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:30.051 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.050+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1a40095f0 con 0x7fd1b00f5330 2024-02-10T09:29:30.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.260+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1b005ea50 con 0x7fd1b00f5330 2024-02-10T09:29:30.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.261+0000 7fd1ad7fa700 1 -- 172.21.15.158:0/2092135228 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd1a40095f0 con 0x7fd1b00f5330 2024-02-10T09:29:30.262 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:30.262 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:30.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd1a002ff60 msgr2=0x7fd1a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd1a002ff60 0x7fd1a0032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fd19c0096a0 tx=0x7fd19c006b40).stop 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 msgr2=0x7fd1b0109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b0109780 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fd1a40089f0 tx=0x7fd1a400b040).stop 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 shutdown_connections 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd1a002ff60 0x7fd1a0032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 --2- 172.21.15.158:0/2092135228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1b00f5330 0x7fd1b0109780 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 >> 172.21.15.158:0/2092135228 conn(0x7fd1b00f0a90 msgr2=0x7fd1b00f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 shutdown_connections 2024-02-10T09:29:30.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:30.262+0000 7fd1b722b700 1 -- 172.21.15.158:0/2092135228 wait complete. 2024-02-10T09:29:30.265 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:30.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:28.102570+0000 mgr.smithi083.dowdym (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:31.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:30 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:30.262121+0000 mon.smithi083 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.158:0/2092135228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:31.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:31.799 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:32.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:30.102838+0000 mgr.smithi083.dowdym (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 -- 172.21.15.158:0/2945091135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 msgr2=0x7f09b00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 --2- 172.21.15.158:0/2945091135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b00f55c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f09ac009fb0 tx=0x7f09ac009b30).stop 2024-02-10T09:29:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 -- 172.21.15.158:0/2945091135 shutdown_connections 2024-02-10T09:29:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 --2- 172.21.15.158:0/2945091135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b00f55c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:33.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 -- 172.21.15.158:0/2945091135 >> 172.21.15.158:0/2945091135 conn(0x7f09b00f09a0 msgr2=0x7f09b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 -- 172.21.15.158:0/2945091135 shutdown_connections 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.785+0000 7f09b62e9700 1 -- 172.21.15.158:0/2945091135 wait complete. 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b62e9700 1 Processor -- start 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b62e9700 1 -- start start 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b62e9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b01092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b62e9700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0060240 con 0x7f09b00f51a0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b52e7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b01092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b52e7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b01092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56830/0 (socket says 172.21.15.158:56830) 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b52e7700 1 -- 172.21.15.158:0/1334298738 learned_addr learned my addr 172.21.15.158:0/1334298738 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b52e7700 1 -- 172.21.15.158:0/1334298738 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac005040 con 0x7f09b00f51a0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.786+0000 7f09b52e7700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b01092d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f09ac009830 tx=0x7f09ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.787+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ac009320 con 0x7f09b00f51a0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.787+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b0060440 con 0x7f09b00f51a0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.787+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09ac013380 con 0x7f09b00f51a0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.787+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ac014420 con 0x7f09b00f51a0 2024-02-10T09:29:33.789 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.787+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0109980 con 0x7f09b00f51a0 2024-02-10T09:29:33.790 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.788+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09ac014580 con 0x7f09b00f51a0 2024-02-10T09:29:33.790 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.788+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b0045ad0 con 0x7f09b00f51a0 2024-02-10T09:29:33.790 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.788+0000 7f09a5ffb700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:33.790 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.788+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09ac040e30 con 0x7f09b00f51a0 2024-02-10T09:29:33.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.788+0000 7f09b4ae6700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:33.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.788+0000 7f09b4ae6700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f09a00096a0 tx=0x7f09a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:33.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:33.793+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09ac01c890 con 0x7f09b00f51a0 2024-02-10T09:29:34.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.000+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09b010a1b0 con 0x7f09b00f51a0 2024-02-10T09:29:34.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.001+0000 7f09a5ffb700 1 -- 172.21.15.158:0/1334298738 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09ac02c2d0 con 0x7f09b00f51a0 2024-02-10T09:29:34.003 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:34.003 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.003+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f099c02ff60 msgr2=0x7f099c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.003+0000 7f09b62e9700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f09a00096a0 tx=0x7f09a0006b40).stop 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.003+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 msgr2=0x7f09b01092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.003+0000 7f09b62e9700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b01092d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f09ac009830 tx=0x7f09ac00b040).stop 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.004+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 shutdown_connections 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.004+0000 7f09b62e9700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.004+0000 7f09b62e9700 1 --2- 172.21.15.158:0/1334298738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09b00f51a0 0x7f09b01092d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:34.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.004+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 >> 172.21.15.158:0/1334298738 conn(0x7f09b00f09a0 msgr2=0x7f09b00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.004+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 shutdown_connections 2024-02-10T09:29:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:34.004+0000 7f09b62e9700 1 -- 172.21.15.158:0/1334298738 wait complete. 2024-02-10T09:29:34.006 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:34.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:32.103122+0000 mgr.smithi083.dowdym (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:35.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:34 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:34.002482+0000 mon.smithi083 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.158:0/1334298738' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:35.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:35.710 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:36.236 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:34.103474+0000 mgr.smithi083.dowdym (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:37.485 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.482+0000 7f2cb404c700 1 -- 172.21.15.158:0/1900688618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 msgr2=0x7f2cac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.482+0000 7f2cb404c700 1 --2- 172.21.15.158:0/1900688618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac0f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2c9c009fb0 tx=0x7f2c9c009b30).stop 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.482+0000 7f2cb404c700 1 -- 172.21.15.158:0/1900688618 shutdown_connections 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.482+0000 7f2cb404c700 1 --2- 172.21.15.158:0/1900688618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac0f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.482+0000 7f2cb404c700 1 -- 172.21.15.158:0/1900688618 >> 172.21.15.158:0/1900688618 conn(0x7f2cac0f1c70 msgr2=0x7f2cac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.483+0000 7f2cb404c700 1 -- 172.21.15.158:0/1900688618 shutdown_connections 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.483+0000 7f2cb404c700 1 -- 172.21.15.158:0/1900688618 wait complete. 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.483+0000 7f2cb404c700 1 Processor -- start 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.483+0000 7f2cb404c700 1 -- start start 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.483+0000 7f2cb404c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.483+0000 7f2cb404c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cac106cd0 con 0x7f2cac0f6510 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb1de8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb1de8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56848/0 (socket says 172.21.15.158:56848) 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb1de8700 1 -- 172.21.15.158:0/182776175 learned_addr learned my addr 172.21.15.158:0/182776175 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:37.486 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb1de8700 1 -- 172.21.15.158:0/182776175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c005040 con 0x7f2cac0f6510 2024-02-10T09:29:37.487 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb1de8700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac1067e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2c9c005010 tx=0x7f2c9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:37.487 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c9c009320 con 0x7f2cac0f6510 2024-02-10T09:29:37.487 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cac106ed0 con 0x7f2cac0f6510 2024-02-10T09:29:37.487 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c9c013380 con 0x7f2cac0f6510 2024-02-10T09:29:37.487 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c9c014e50 con 0x7f2cac0f6510 2024-02-10T09:29:37.487 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.484+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cac107360 con 0x7f2cac0f6510 2024-02-10T09:29:37.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.485+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c9c0095f0 con 0x7f2cac0f6510 2024-02-10T09:29:37.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.485+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cac045ad0 con 0x7f2cac0f6510 2024-02-10T09:29:37.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.485+0000 7f2ca27fc700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2c9802ff10 0x7f2c980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:37.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.485+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c9c040f10 con 0x7f2cac0f6510 2024-02-10T09:29:37.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.485+0000 7f2cb15e7700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2c9802ff10 0x7f2c980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:37.488 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.486+0000 7f2cb15e7700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2c9802ff10 0x7f2c980323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2ca80096a0 tx=0x7f2ca8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:37.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.490+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c9c014420 con 0x7f2cac0f6510 2024-02-10T09:29:37.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.693+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cac108210 con 0x7f2cac0f6510 2024-02-10T09:29:37.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.694+0000 7f2ca27fc700 1 -- 172.21.15.158:0/182776175 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c9c010230 con 0x7f2cac0f6510 2024-02-10T09:29:37.695 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:37.695 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.695+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2c9802ff10 msgr2=0x7f2c980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.695+0000 7f2cb404c700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2c9802ff10 0x7f2c980323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f2ca80096a0 tx=0x7f2ca8006b40).stop 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.695+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 msgr2=0x7f2cac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.695+0000 7f2cb404c700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac1067e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f2c9c005010 tx=0x7f2c9c00b040).stop 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.696+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 shutdown_connections 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.696+0000 7f2cb404c700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2c9802ff10 0x7f2c980323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.696+0000 7f2cb404c700 1 --2- 172.21.15.158:0/182776175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cac0f6510 0x7f2cac1067e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.696+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 >> 172.21.15.158:0/182776175 conn(0x7f2cac0f1c70 msgr2=0x7f2cac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:37.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.696+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 shutdown_connections 2024-02-10T09:29:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:37.696+0000 7f2cb404c700 1 -- 172.21.15.158:0/182776175 wait complete. 2024-02-10T09:29:37.698 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:38.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:36.103833+0000 mgr.smithi083.dowdym (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:37.695432+0000 mon.smithi083 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.158:0/182776175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:39.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:39.363 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:40.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:38.104271+0000 mgr.smithi083.dowdym (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:41.283 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 -- 172.21.15.158:0/1073850598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 msgr2=0x7f5a780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 --2- 172.21.15.158:0/1073850598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a780f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5a68009fb0 tx=0x7f5a68009b30).stop 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 -- 172.21.15.158:0/1073850598 shutdown_connections 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 --2- 172.21.15.158:0/1073850598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a780f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 -- 172.21.15.158:0/1073850598 >> 172.21.15.158:0/1073850598 conn(0x7f5a780f0b70 msgr2=0x7f5a780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 -- 172.21.15.158:0/1073850598 shutdown_connections 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 -- 172.21.15.158:0/1073850598 wait complete. 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 Processor -- start 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.230+0000 7f5a7e783700 1 -- start start 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a7e783700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a781094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a7e783700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a78060240 con 0x7f5a780f5410 2024-02-10T09:29:41.284 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a77fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a781094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a77fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a781094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53042/0 (socket says 172.21.15.158:53042) 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a77fff700 1 -- 172.21.15.158:0/1914619506 learned_addr learned my addr 172.21.15.158:0/1914619506 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a77fff700 1 -- 172.21.15.158:0/1914619506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a68005040 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a77fff700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a781094e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f5a68005010 tx=0x7f5a6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a68009320 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a78060440 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a68013380 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a68014e50 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.231+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a78109b90 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.232+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a680095f0 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.232+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a78045ad0 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.232+0000 7f5a757fa700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5a6402ff10 0x7f5a640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.232+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a68040fa0 con 0x7f5a780f5410 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.233+0000 7f5a6ffff700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5a6402ff10 0x7f5a640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.233+0000 7f5a6ffff700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5a6402ff10 0x7f5a640323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5a600096a0 tx=0x7f5a60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:41.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.237+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a68014420 con 0x7f5a780f5410 2024-02-10T09:29:41.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.441+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a7810a3b0 con 0x7f5a780f5410 2024-02-10T09:29:41.443 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.442+0000 7f5a757fa700 1 -- 172.21.15.158:0/1914619506 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a68010230 con 0x7f5a780f5410 2024-02-10T09:29:41.443 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:41.443 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5a6402ff10 msgr2=0x7f5a640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5a6402ff10 0x7f5a640323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5a600096a0 tx=0x7f5a60006b40).stop 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 msgr2=0x7f5a781094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a781094e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f5a68005010 tx=0x7f5a6800b040).stop 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 shutdown_connections 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5a6402ff10 0x7f5a640323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 --2- 172.21.15.158:0/1914619506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a780f5410 0x7f5a781094e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 >> 172.21.15.158:0/1914619506 conn(0x7f5a780f0b70 msgr2=0x7f5a780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:41.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.443+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 shutdown_connections 2024-02-10T09:29:41.446 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:41.444+0000 7f5a7e783700 1 -- 172.21.15.158:0/1914619506 wait complete. 2024-02-10T09:29:41.446 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:40.104692+0000 mgr.smithi083.dowdym (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:41.443270+0000 mon.smithi083 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.158:0/1914619506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:43.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:43.125 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:44.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:42.105016+0000 mgr.smithi083.dowdym (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:44.971 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 -- 172.21.15.158:0/1439641884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 msgr2=0x7f51040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:44.972 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 --2- 172.21.15.158:0/1439641884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51040f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f50f8009fb0 tx=0x7f50f8009b30).stop 2024-02-10T09:29:44.972 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 -- 172.21.15.158:0/1439641884 shutdown_connections 2024-02-10T09:29:44.972 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 --2- 172.21.15.158:0/1439641884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51040f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:44.972 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 -- 172.21.15.158:0/1439641884 >> 172.21.15.158:0/1439641884 conn(0x7f51040f1c70 msgr2=0x7f51040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:44.972 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 -- 172.21.15.158:0/1439641884 shutdown_connections 2024-02-10T09:29:44.972 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.970+0000 7f510963b700 1 -- 172.21.15.158:0/1439641884 wait complete. 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.971+0000 7f510963b700 1 Processor -- start 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.971+0000 7f510963b700 1 -- start start 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.971+0000 7f510963b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51041067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.971+0000 7f510963b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5104106cd0 con 0x7f51040f6510 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f5102ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51041067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f5102ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51041067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53060/0 (socket says 172.21.15.158:53060) 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f5102ffd700 1 -- 172.21.15.158:0/1728373775 learned_addr learned my addr 172.21.15.158:0/1728373775 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f5102ffd700 1 -- 172.21.15.158:0/1728373775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f8005040 con 0x7f51040f6510 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f5102ffd700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51041067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f50f8005010 tx=0x7f50f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50f8009320 con 0x7f51040f6510 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5104106ed0 con 0x7f51040f6510 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50f8013380 con 0x7f51040f6510 2024-02-10T09:29:44.974 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50f8014e50 con 0x7f51040f6510 2024-02-10T09:29:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.972+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5104107360 con 0x7f51040f6510 2024-02-10T09:29:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.973+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50f80263a0 con 0x7f51040f6510 2024-02-10T09:29:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.973+0000 7f50eb7fe700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f50ec02ff10 0x7f50ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:44.975 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.973+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50f8040f80 con 0x7f51040f6510 2024-02-10T09:29:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.973+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5104045ad0 con 0x7f51040f6510 2024-02-10T09:29:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.973+0000 7f51027fc700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f50ec02ff10 0x7f50ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:44.976 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.974+0000 7f51027fc700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f50ec02ff10 0x7f50ec0323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f50f40096a0 tx=0x7f50f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:44.979 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:44.978+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50f8012920 con 0x7f51040f6510 2024-02-10T09:29:45.184 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.183+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51040fc0a0 con 0x7f51040f6510 2024-02-10T09:29:45.185 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.184+0000 7f50eb7fe700 1 -- 172.21.15.158:0/1728373775 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50f802b020 con 0x7f51040f6510 2024-02-10T09:29:45.185 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:45.185 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f50ec02ff10 msgr2=0x7f50ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f50ec02ff10 0x7f50ec0323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f50f40096a0 tx=0x7f50f4006b40).stop 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 msgr2=0x7f51041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51041067e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f50f8005010 tx=0x7f50f800b040).stop 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 shutdown_connections 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f50ec02ff10 0x7f50ec0323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 --2- 172.21.15.158:0/1728373775 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51040f6510 0x7f51041067e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.185+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 >> 172.21.15.158:0/1728373775 conn(0x7f51040f1c70 msgr2=0x7f51040f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:45.187 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.186+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 shutdown_connections 2024-02-10T09:29:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:45.186+0000 7f510963b700 1 -- 172.21.15.158:0/1728373775 wait complete. 2024-02-10T09:29:45.188 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:44.105350+0000 mgr.smithi083.dowdym (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:46.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:29:44.565317+0000 mgr.smithi083.dowdym (mgr.14162) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi083 ... 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Verifying port 9100 ... 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 1 from systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: systemctl: stderr Job for ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083.service failed because a timeout was exceeded. 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: systemctl: stderr See "systemctl status ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083.service" and "journalctl -xe" for details. 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: main() 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: r = ctx.func(ctx) 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: return func(ctx) 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ports=daemon_ports) 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-10T09:29:46.277 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: RuntimeError: Failed command: systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: yield (conn, connr) 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: code, '\n'.join(err))) 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi083 ... 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Verifying port 9100 ... 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 1 from systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: systemctl: stderr Job for ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083.service failed because a timeout was exceeded. 2024-02-10T09:29:46.278 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: systemctl: stderr See "systemctl status ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083.service" and "journalctl -xe" for details. 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: main() 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: r = ctx.func(ctx) 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: return func(ctx) 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: ports=daemon_ports) 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-10T09:29:46.279 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: RuntimeError: Failed command: systemctl start ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@node-exporter.smithi083 2024-02-10T09:29:46.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:29:44.571096+0000 mgr.smithi083.dowdym (mgr.14162) 233 : cephadm [INF] Deploying daemon prometheus.smithi083 on smithi083 2024-02-10T09:29:46.280 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:45.185134+0000 mon.smithi083 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.158:0/1728373775' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:46.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:46.889 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:46.105694+0000 mgr.smithi083.dowdym (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 -- 172.21.15.158:0/153009890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 msgr2=0x7fc6600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 --2- 172.21.15.158:0/153009890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6600f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fc654009fb0 tx=0x7fc654009b30).stop 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 -- 172.21.15.158:0/153009890 shutdown_connections 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 --2- 172.21.15.158:0/153009890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6600f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 -- 172.21.15.158:0/153009890 >> 172.21.15.158:0/153009890 conn(0x7fc6600f0b70 msgr2=0x7fc6600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 -- 172.21.15.158:0/153009890 shutdown_connections 2024-02-10T09:29:48.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.736+0000 7fc665697700 1 -- 172.21.15.158:0/153009890 wait complete. 2024-02-10T09:29:48.739 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.737+0000 7fc665697700 1 Processor -- start 2024-02-10T09:29:48.739 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.737+0000 7fc665697700 1 -- start start 2024-02-10T09:29:48.740 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.737+0000 7fc665697700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6601094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:48.740 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.737+0000 7fc665697700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc660060280 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc65effd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6601094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc65effd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6601094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53070/0 (socket says 172.21.15.158:53070) 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc65effd700 1 -- 172.21.15.158:0/3171556912 learned_addr learned my addr 172.21.15.158:0/3171556912 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc65effd700 1 -- 172.21.15.158:0/3171556912 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc654005040 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc65effd700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6601094e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc6540089f0 tx=0x7fc65400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc654009320 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc654013380 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc660060480 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65401ed80 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.738+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc660109c10 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.739+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc65401e420 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.739+0000 7fc6477fe700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc64802ff60 0x7fc648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.739+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc660045ad0 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.739+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc654041150 con 0x7fc6600f5410 2024-02-10T09:29:48.741 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.739+0000 7fc65e7fc700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc64802ff60 0x7fc648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:48.742 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.740+0000 7fc65e7fc700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc64802ff60 0x7fc648032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc6500096a0 tx=0x7fc650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:48.745 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.744+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc65401d380 con 0x7fc6600f5410 2024-02-10T09:29:48.950 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.949+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6600fafa0 con 0x7fc6600f5410 2024-02-10T09:29:48.951 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.950+0000 7fc6477fe700 1 -- 172.21.15.158:0/3171556912 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc654004490 con 0x7fc6600f5410 2024-02-10T09:29:48.951 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:48.951 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc64802ff60 msgr2=0x7fc648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc64802ff60 0x7fc648032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc6500096a0 tx=0x7fc650006b40).stop 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 msgr2=0x7fc6601094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6601094e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc6540089f0 tx=0x7fc65400b040).stop 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 shutdown_connections 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc64802ff60 0x7fc648032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 --2- 172.21.15.158:0/3171556912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6600f5410 0x7fc6601094e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.951+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 >> 172.21.15.158:0/3171556912 conn(0x7fc6600f0b70 msgr2=0x7fc6600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:48.953 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.952+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 shutdown_connections 2024-02-10T09:29:48.954 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:48.952+0000 7fc665697700 1 -- 172.21.15.158:0/3171556912 wait complete. 2024-02-10T09:29:48.954 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:48.106068+0000 mgr.smithi083.dowdym (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:48.951315+0000 mon.smithi083 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.158:0/3171556912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:50.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:50.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:52.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:50.106413+0000 mgr.smithi083.dowdym (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 -- 172.21.15.158:0/257499991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 msgr2=0x7f84780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 --2- 172.21.15.158:0/257499991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f8460009fb0 tx=0x7f8460009b30).stop 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 -- 172.21.15.158:0/257499991 shutdown_connections 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 --2- 172.21.15.158:0/257499991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 -- 172.21.15.158:0/257499991 >> 172.21.15.158:0/257499991 conn(0x7f84780f0b70 msgr2=0x7f84780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 -- 172.21.15.158:0/257499991 shutdown_connections 2024-02-10T09:29:52.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.373+0000 7f847d466700 1 -- 172.21.15.158:0/257499991 wait complete. 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f847d466700 1 Processor -- start 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f847d466700 1 -- start start 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f847d466700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f847d466700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84780fb960 con 0x7f84780f5410 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f8476ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f8476ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53602/0 (socket says 172.21.15.158:53602) 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.374+0000 7f8476ffd700 1 -- 172.21.15.158:0/1960125057 learned_addr learned my addr 172.21.15.158:0/1960125057 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f8476ffd700 1 -- 172.21.15.158:0/1960125057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8460005040 con 0x7f84780f5410 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f8476ffd700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780fd4e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f8460005010 tx=0x7f846000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8460009320 con 0x7f84780f5410 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84780fbb00 con 0x7f84780f5410 2024-02-10T09:29:52.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8460013380 con 0x7f84780f5410 2024-02-10T09:29:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8460014e50 con 0x7f84780f5410 2024-02-10T09:29:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.375+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84780fbf90 con 0x7f84780f5410 2024-02-10T09:29:52.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.376+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84600263a0 con 0x7f84780f5410 2024-02-10T09:29:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.376+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8478059100 con 0x7f84780f5410 2024-02-10T09:29:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.376+0000 7f846f7fe700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f846402ff10 0x7f84640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.376+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8460041180 con 0x7f84780f5410 2024-02-10T09:29:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.376+0000 7f84767fc700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f846402ff10 0x7f84640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:52.379 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.377+0000 7f84767fc700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f846402ff10 0x7f84640323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f84680096a0 tx=0x7f8468006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:52.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.380+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8460014420 con 0x7f84780f5410 2024-02-10T09:29:52.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.585+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84780fafa0 con 0x7f84780f5410 2024-02-10T09:29:52.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.586+0000 7f846f7fe700 1 -- 172.21.15.158:0/1960125057 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f846000f340 con 0x7f84780f5410 2024-02-10T09:29:52.587 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:52.588 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:52.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f846402ff10 msgr2=0x7f84640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:52.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f846402ff10 0x7f84640323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f84680096a0 tx=0x7f8468006b40).stop 2024-02-10T09:29:52.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 msgr2=0x7f84780fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:52.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780fd4e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f8460005010 tx=0x7f846000b040).stop 2024-02-10T09:29:52.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 shutdown_connections 2024-02-10T09:29:52.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f846402ff10 0x7f84640323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:52.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 --2- 172.21.15.158:0/1960125057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84780f5410 0x7f84780fd4e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:52.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 >> 172.21.15.158:0/1960125057 conn(0x7f84780f0b70 msgr2=0x7f847805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:52.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 shutdown_connections 2024-02-10T09:29:52.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:52.588+0000 7f847d466700 1 -- 172.21.15.158:0/1960125057 wait complete. 2024-02-10T09:29:52.590 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:53.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:52 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:52.587549+0000 mon.smithi083 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.158:0/1960125057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:54.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:54.198 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:54.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:52.106777+0000 mgr.smithi083.dowdym (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:56.117 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 -- 172.21.15.158:0/1109254080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 msgr2=0x7fe6180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 --2- 172.21.15.158:0/1109254080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6180f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fe60c009fb0 tx=0x7fe60c009b30).stop 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 -- 172.21.15.158:0/1109254080 shutdown_connections 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 --2- 172.21.15.158:0/1109254080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6180f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 -- 172.21.15.158:0/1109254080 >> 172.21.15.158:0/1109254080 conn(0x7fe6180f1c70 msgr2=0x7fe6180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 -- 172.21.15.158:0/1109254080 shutdown_connections 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.115+0000 7fe61da60700 1 -- 172.21.15.158:0/1109254080 wait complete. 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.116+0000 7fe61da60700 1 Processor -- start 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.116+0000 7fe61da60700 1 -- start start 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.116+0000 7fe61da60700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6181067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:56.118 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.116+0000 7fe61da60700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe618106c90 con 0x7fe6180f6510 2024-02-10T09:29:56.120 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.116+0000 7fe616ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6181067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:56.120 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe616ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6181067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53618/0 (socket says 172.21.15.158:53618) 2024-02-10T09:29:56.120 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe616ffd700 1 -- 172.21.15.158:0/4002487041 learned_addr learned my addr 172.21.15.158:0/4002487041 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe616ffd700 1 -- 172.21.15.158:0/4002487041 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c005040 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe616ffd700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6181067a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fe60c005010 tx=0x7fe60c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe60c009320 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe618106e90 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe60c013380 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe60c014e50 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.117+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618107320 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.118+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe60c0095f0 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.118+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe618045ad0 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.118+0000 7fe5ff7fe700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe60002ff10 0x7fe6000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.118+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe60c040fa0 con 0x7fe6180f6510 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.118+0000 7fe6167fc700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe60002ff10 0x7fe6000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:29:56.121 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.119+0000 7fe6167fc700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe60002ff10 0x7fe6000323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe6080096a0 tx=0x7fe608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:29:56.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.123+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe60c014420 con 0x7fe6180f6510 2024-02-10T09:29:56.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:54.107128+0000 mgr.smithi083.dowdym (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:29:56.330 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.329+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6181081d0 con 0x7fe6180f6510 2024-02-10T09:29:56.331 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.330+0000 7fe5ff7fe700 1 -- 172.21.15.158:0/4002487041 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe60c010230 con 0x7fe6180f6510 2024-02-10T09:29:56.331 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:29:56.331 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.331+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe60002ff10 msgr2=0x7fe6000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.331+0000 7fe61da60700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe60002ff10 0x7fe6000323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fe6080096a0 tx=0x7fe608006b40).stop 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.331+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 msgr2=0x7fe6181067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.331+0000 7fe61da60700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6181067a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fe60c005010 tx=0x7fe60c00b040).stop 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.332+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 shutdown_connections 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.332+0000 7fe61da60700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe60002ff10 0x7fe6000323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.332+0000 7fe61da60700 1 --2- 172.21.15.158:0/4002487041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe6180f6510 0x7fe6181067a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.332+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 >> 172.21.15.158:0/4002487041 conn(0x7fe6180f1c70 msgr2=0x7fe6180f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.332+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 shutdown_connections 2024-02-10T09:29:56.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:29:56.332+0000 7fe61da60700 1 -- 172.21.15.158:0/4002487041 wait complete. 2024-02-10T09:29:56.334 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:29:57.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:56 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:29:56.331317+0000 mon.smithi083 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.158:0/4002487041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:29:58.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:29:58.044 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:29:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:56.107509+0000 mgr.smithi083.dowdym (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:00.074 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 -- 172.21.15.158:0/3157028899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 msgr2=0x7fe8380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:00.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 --2- 172.21.15.158:0/3157028899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe8380f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fe828009fb0 tx=0x7fe828009b30).stop 2024-02-10T09:30:00.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 -- 172.21.15.158:0/3157028899 shutdown_connections 2024-02-10T09:30:00.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 --2- 172.21.15.158:0/3157028899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe8380f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:00.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 -- 172.21.15.158:0/3157028899 >> 172.21.15.158:0/3157028899 conn(0x7fe8380f1c70 msgr2=0x7fe8380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:00.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 -- 172.21.15.158:0/3157028899 shutdown_connections 2024-02-10T09:30:00.075 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.073+0000 7fe83d37b700 1 -- 172.21.15.158:0/3157028899 wait complete. 2024-02-10T09:30:00.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe83d37b700 1 Processor -- start 2024-02-10T09:30:00.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe83d37b700 1 -- start start 2024-02-10T09:30:00.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe83d37b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe83806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:00.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe83d37b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838069e20 con 0x7fe8380f6510 2024-02-10T09:30:00.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe836ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe83806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:00.076 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe836ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe83806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53486/0 (socket says 172.21.15.158:53486) 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe836ffd700 1 -- 172.21.15.158:0/254882175 learned_addr learned my addr 172.21.15.158:0/254882175 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe836ffd700 1 -- 172.21.15.158:0/254882175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe828005040 con 0x7fe8380f6510 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.074+0000 7fe836ffd700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe83806b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe828005010 tx=0x7fe82800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe828009320 con 0x7fe8380f6510 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe838069fc0 con 0x7fe8380f6510 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe828013380 con 0x7fe8380f6510 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe828014e50 con 0x7fe8380f6510 2024-02-10T09:30:00.077 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83806a450 con 0x7fe8380f6510 2024-02-10T09:30:00.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8280095f0 con 0x7fe8380f6510 2024-02-10T09:30:00.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.075+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe838059100 con 0x7fe8380f6510 2024-02-10T09:30:00.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.076+0000 7fe81f7fe700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe82002ff10 0x7fe8200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:00.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.076+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe828040f60 con 0x7fe8380f6510 2024-02-10T09:30:00.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.076+0000 7fe8367fc700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe82002ff10 0x7fe8200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:00.078 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.076+0000 7fe8367fc700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe82002ff10 0x7fe8200323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe82c0096a0 tx=0x7fe82c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:00.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.080+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe828014420 con 0x7fe8380f6510 2024-02-10T09:30:00.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:29:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:29:58.107855+0000 mgr.smithi083.dowdym (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:00.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.289+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8380fc0a0 con 0x7fe8380f6510 2024-02-10T09:30:00.291 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.290+0000 7fe81f7fe700 1 -- 172.21.15.158:0/254882175 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe828010230 con 0x7fe8380f6510 2024-02-10T09:30:00.291 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:00.291 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe82002ff10 msgr2=0x7fe8200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe82002ff10 0x7fe8200323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe82c0096a0 tx=0x7fe82c006b40).stop 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 msgr2=0x7fe83806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe83806b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe828005010 tx=0x7fe82800b040).stop 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 shutdown_connections 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe82002ff10 0x7fe8200323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 --2- 172.21.15.158:0/254882175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8380f6510 0x7fe83806b9a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 >> 172.21.15.158:0/254882175 conn(0x7fe8380f1c70 msgr2=0x7fe8380f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 shutdown_connections 2024-02-10T09:30:00.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:00.291+0000 7fe83d37b700 1 -- 172.21.15.158:0/254882175 wait complete. 2024-02-10T09:30:00.294 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:01.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:00 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:00.291103+0000 mon.smithi083 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.158:0/254882175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:02.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:02.021 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:02.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:00.108196+0000 mgr.smithi083.dowdym (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 -- 172.21.15.158:0/1889695593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 msgr2=0x7f71140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 --2- 172.21.15.158:0/1889695593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f71140f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f7104009fb0 tx=0x7f7104009b30).stop 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 -- 172.21.15.158:0/1889695593 shutdown_connections 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 --2- 172.21.15.158:0/1889695593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f71140f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 -- 172.21.15.158:0/1889695593 >> 172.21.15.158:0/1889695593 conn(0x7f71140f0b70 msgr2=0x7f71140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 -- 172.21.15.158:0/1889695593 shutdown_connections 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 -- 172.21.15.158:0/1889695593 wait complete. 2024-02-10T09:30:04.067 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 Processor -- start 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.064+0000 7f7119696700 1 -- start start 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7119696700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f7114109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7119696700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7114060240 con 0x7f71140f5410 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7112ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f7114109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7112ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f7114109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53496/0 (socket says 172.21.15.158:53496) 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7112ffd700 1 -- 172.21.15.158:0/140057904 learned_addr learned my addr 172.21.15.158:0/140057904 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7112ffd700 1 -- 172.21.15.158:0/140057904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7104005040 con 0x7f71140f5410 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f7112ffd700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f7114109520 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7104005010 tx=0x7f710400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.065+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7104009320 con 0x7f71140f5410 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.066+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7114060440 con 0x7f71140f5410 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.066+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7104013380 con 0x7f71140f5410 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.066+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7104014e50 con 0x7f71140f5410 2024-02-10T09:30:04.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.066+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7114109bd0 con 0x7f71140f5410 2024-02-10T09:30:04.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.066+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71040095f0 con 0x7f71140f5410 2024-02-10T09:30:04.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.066+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7114045ad0 con 0x7f71140f5410 2024-02-10T09:30:04.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.067+0000 7f70fb7fe700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70fc02ff10 0x7f70fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:04.070 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.067+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7104040fa0 con 0x7f71140f5410 2024-02-10T09:30:04.070 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.067+0000 7f71127fc700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70fc02ff10 0x7f70fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:04.070 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.067+0000 7f71127fc700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70fc02ff10 0x7f70fc0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f71080096a0 tx=0x7f7108006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:04.072 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.071+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7104014420 con 0x7f71140f5410 2024-02-10T09:30:04.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:02.108482+0000 mgr.smithi083.dowdym (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:04.277 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.276+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f711410a3f0 con 0x7f71140f5410 2024-02-10T09:30:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.276+0000 7f70fb7fe700 1 -- 172.21.15.158:0/140057904 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7104010230 con 0x7f71140f5410 2024-02-10T09:30:04.278 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:04.278 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70fc02ff10 msgr2=0x7f70fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70fc02ff10 0x7f70fc0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f71080096a0 tx=0x7f7108006b40).stop 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 msgr2=0x7f7114109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f7114109520 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7104005010 tx=0x7f710400b040).stop 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 shutdown_connections 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70fc02ff10 0x7f70fc0323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 --2- 172.21.15.158:0/140057904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f71140f5410 0x7f7114109520 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 >> 172.21.15.158:0/140057904 conn(0x7f71140f0b70 msgr2=0x7f71140f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 shutdown_connections 2024-02-10T09:30:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:04.278+0000 7f7119696700 1 -- 172.21.15.158:0/140057904 wait complete. 2024-02-10T09:30:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:05.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:04.277981+0000 mon.smithi083 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.158:0/140057904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:06.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:06.028 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:06.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:04.108805+0000 mgr.smithi083.dowdym (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 -- 172.21.15.158:0/3159580439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 msgr2=0x7f17dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 --2- 172.21.15.158:0/3159580439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc0f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f17cc009fb0 tx=0x7f17cc009b30).stop 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 -- 172.21.15.158:0/3159580439 shutdown_connections 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 --2- 172.21.15.158:0/3159580439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc0f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 -- 172.21.15.158:0/3159580439 >> 172.21.15.158:0/3159580439 conn(0x7f17dc0f0b70 msgr2=0x7f17dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 -- 172.21.15.158:0/3159580439 shutdown_connections 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.916+0000 7f17e268e700 1 -- 172.21.15.158:0/3159580439 wait complete. 2024-02-10T09:30:07.930 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.917+0000 7f17e268e700 1 Processor -- start 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.917+0000 7f17e268e700 1 -- start start 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.917+0000 7f17e268e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.917+0000 7f17e268e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc060240 con 0x7f17dc0f5410 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.917+0000 7f17dbfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17dbfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53520/0 (socket says 172.21.15.158:53520) 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17dbfff700 1 -- 172.21.15.158:0/167691834 learned_addr learned my addr 172.21.15.158:0/167691834 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17dbfff700 1 -- 172.21.15.158:0/167691834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc005040 con 0x7f17dc0f5410 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17dbfff700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc109510 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f17cc005010 tx=0x7f17cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17cc009320 con 0x7f17dc0f5410 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc060440 con 0x7f17dc0f5410 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17cc013380 con 0x7f17dc0f5410 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17cc014e50 con 0x7f17dc0f5410 2024-02-10T09:30:07.931 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.918+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc109bc0 con 0x7f17dc0f5410 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.919+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17cc014420 con 0x7f17dc0f5410 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.919+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc045ad0 con 0x7f17dc0f5410 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.919+0000 7f17d8ff9700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17c402ff10 0x7f17c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.919+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17cc040e30 con 0x7f17dc0f5410 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.920+0000 7f17db7fe700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17c402ff10 0x7f17c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.920+0000 7f17db7fe700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17c402ff10 0x7f17c40323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f17d00096a0 tx=0x7f17d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:07.932 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:07.924+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17cc0095f0 con 0x7f17dc0f5410 2024-02-10T09:30:08.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.126+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17dc10a3e0 con 0x7f17dc0f5410 2024-02-10T09:30:08.128 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.127+0000 7f17d8ff9700 1 -- 172.21.15.158:0/167691834 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f17cc01db70 con 0x7f17dc0f5410 2024-02-10T09:30:08.128 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:08.128 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17c402ff10 msgr2=0x7f17c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17c402ff10 0x7f17c40323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f17d00096a0 tx=0x7f17d0006b40).stop 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 msgr2=0x7f17dc109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc109510 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f17cc005010 tx=0x7f17cc00b040).stop 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 shutdown_connections 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f17c402ff10 0x7f17c40323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 --2- 172.21.15.158:0/167691834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc0f5410 0x7f17dc109510 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 >> 172.21.15.158:0/167691834 conn(0x7f17dc0f0b70 msgr2=0x7f17dc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:08.130 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 shutdown_connections 2024-02-10T09:30:08.131 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:08.128+0000 7f17e268e700 1 -- 172.21.15.158:0/167691834 wait complete. 2024-02-10T09:30:08.131 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:08.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:06.109178+0000 mgr.smithi083.dowdym (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:09.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:08.127921+0000 mon.smithi083 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.158:0/167691834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:09.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:09.884 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:10.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:08.109546+0000 mgr.smithi083.dowdym (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:11.921 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 -- 172.21.15.158:0/3527280934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 msgr2=0x7f7f340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:11.921 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 --2- 172.21.15.158:0/3527280934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f90f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f7f28009fb0 tx=0x7f7f28009b30).stop 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 -- 172.21.15.158:0/3527280934 shutdown_connections 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 --2- 172.21.15.158:0/3527280934 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f90f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 -- 172.21.15.158:0/3527280934 >> 172.21.15.158:0/3527280934 conn(0x7f7f340f0b70 msgr2=0x7f7f340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 -- 172.21.15.158:0/3527280934 shutdown_connections 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.919+0000 7f7f3882d700 1 -- 172.21.15.158:0/3527280934 wait complete. 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.920+0000 7f7f3882d700 1 Processor -- start 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.920+0000 7f7f3882d700 1 -- start start 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.920+0000 7f7f3882d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.920+0000 7f7f3882d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f340f4ff0 con 0x7f7f34060280 2024-02-10T09:30:11.923 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3259c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:45726/0 (socket says 172.21.15.158:45726) 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3259c700 1 -- 172.21.15.158:0/2257895473 learned_addr learned my addr 172.21.15.158:0/2257895473 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3259c700 1 -- 172.21.15.158:0/2257895473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f28005040 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3259c700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f6b70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f7f280089f0 tx=0x7f7f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f28009320 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f340f5190 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f28013380 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f28014e20 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.921+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f340f5620 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.922+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f280095f0 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.922+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f34059100 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.922+0000 7f7f22ffd700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7f1c02ff10 0x7f7f1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.922+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f28040e90 con 0x7f7f34060280 2024-02-10T09:30:11.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.922+0000 7f7f31d9b700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7f1c02ff10 0x7f7f1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.923+0000 7f7f31d9b700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7f1c02ff10 0x7f7f1c0323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7f240096a0 tx=0x7f7f24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:11.928 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:11.927+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f28014420 con 0x7f7f34060280 2024-02-10T09:30:12.138 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.137+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f340fc090 con 0x7f7f34060280 2024-02-10T09:30:12.138 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.137+0000 7f7f22ffd700 1 -- 172.21.15.158:0/2257895473 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f2801d980 con 0x7f7f34060280 2024-02-10T09:30:12.139 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:12.139 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:12.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7f1c02ff10 msgr2=0x7f7f1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:12.140 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7f1c02ff10 0x7f7f1c0323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7f240096a0 tx=0x7f7f24006b40).stop 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 msgr2=0x7f7f340f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f6b70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f7f280089f0 tx=0x7f7f2800b040).stop 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 shutdown_connections 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f7f1c02ff10 0x7f7f1c0323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 --2- 172.21.15.158:0/2257895473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f34060280 0x7f7f340f6b70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 >> 172.21.15.158:0/2257895473 conn(0x7f7f340f0b70 msgr2=0x7f7f340f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 shutdown_connections 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:12.139+0000 7f7f3882d700 1 -- 172.21.15.158:0/2257895473 wait complete. 2024-02-10T09:30:12.141 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:10.109928+0000 mgr.smithi083.dowdym (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:13.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:12 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:12.138726+0000 mon.smithi083 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.158:0/2257895473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:13.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:13.749 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:14.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:12.110296+0000 mgr.smithi083.dowdym (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:15.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:14.173709+0000 mon.smithi083 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:30:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:14.181636+0000 mon.smithi083 (mon.0) 260 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.658+0000 7f0370305700 1 -- 172.21.15.158:0/6465193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 msgr2=0x7f03680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.658+0000 7f0370305700 1 --2- 172.21.15.158:0/6465193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f03680f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f0358009fb0 tx=0x7f0358009b30).stop 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 -- 172.21.15.158:0/6465193 shutdown_connections 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 --2- 172.21.15.158:0/6465193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f03680f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 -- 172.21.15.158:0/6465193 >> 172.21.15.158:0/6465193 conn(0x7f03680f1c70 msgr2=0x7f03680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 -- 172.21.15.158:0/6465193 shutdown_connections 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 -- 172.21.15.158:0/6465193 wait complete. 2024-02-10T09:30:15.661 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 Processor -- start 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 -- start start 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f0368108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.659+0000 7f0370305700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0368108ef0 con 0x7f03680f6510 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f036e0a1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f0368108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f036e0a1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f0368108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:45742/0 (socket says 172.21.15.158:45742) 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f036e0a1700 1 -- 172.21.15.158:0/3091023922 learned_addr learned my addr 172.21.15.158:0/3091023922 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f036e0a1700 1 -- 172.21.15.158:0/3091023922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0358005040 con 0x7f03680f6510 2024-02-10T09:30:15.662 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f036e0a1700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f0368108a00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f0358005010 tx=0x7f035800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:15.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0358009320 con 0x7f03680f6510 2024-02-10T09:30:15.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03681090f0 con 0x7f03680f6510 2024-02-10T09:30:15.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0358013380 con 0x7f03680f6510 2024-02-10T09:30:15.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0358014e50 con 0x7f03680f6510 2024-02-10T09:30:15.663 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.660+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0368109580 con 0x7f03680f6510 2024-02-10T09:30:15.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.661+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03580095f0 con 0x7f03680f6510 2024-02-10T09:30:15.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.661+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0368045ad0 con 0x7f03680f6510 2024-02-10T09:30:15.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.661+0000 7f035effd700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f035402ff10 0x7f03540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:15.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.661+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0358040e90 con 0x7f03680f6510 2024-02-10T09:30:15.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.661+0000 7f036d8a0700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f035402ff10 0x7f03540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:15.664 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.662+0000 7f036d8a0700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f035402ff10 0x7f03540323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f03640096a0 tx=0x7f0364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:15.667 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.666+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0358014420 con 0x7f03680f6510 2024-02-10T09:30:15.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.870+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03680fc0a0 con 0x7f03680f6510 2024-02-10T09:30:15.872 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.871+0000 7f035effd700 1 -- 172.21.15.158:0/3091023922 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0358010230 con 0x7f03680f6510 2024-02-10T09:30:15.872 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:15.872 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:15.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f035402ff10 msgr2=0x7f03540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f035402ff10 0x7f03540323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f03640096a0 tx=0x7f0364006b40).stop 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 msgr2=0x7f0368108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f0368108a00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f0358005010 tx=0x7f035800b040).stop 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 shutdown_connections 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f035402ff10 0x7f03540323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 --2- 172.21.15.158:0/3091023922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03680f6510 0x7f0368108a00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 >> 172.21.15.158:0/3091023922 conn(0x7f03680f1c70 msgr2=0x7f03680f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 shutdown_connections 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:15.872+0000 7f0370305700 1 -- 172.21.15.158:0/3091023922 wait complete. 2024-02-10T09:30:15.874 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:16.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:14.110610+0000 mgr.smithi083.dowdym (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:15.872029+0000 mon.smithi083 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.158:0/3091023922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:17.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:17.663 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:16.110893+0000 mgr.smithi083.dowdym (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:19.521 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 -- 172.21.15.158:0/1132249194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 msgr2=0x7f57000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:19.522 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 --2- 172.21.15.158:0/1132249194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57000f6930 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f56f4009fb0 tx=0x7f56f4009b30).stop 2024-02-10T09:30:19.522 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 -- 172.21.15.158:0/1132249194 shutdown_connections 2024-02-10T09:30:19.522 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 --2- 172.21.15.158:0/1132249194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57000f6930 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:19.522 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 -- 172.21.15.158:0/1132249194 >> 172.21.15.158:0/1132249194 conn(0x7f57000f1c70 msgr2=0x7f57000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:19.522 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 -- 172.21.15.158:0/1132249194 shutdown_connections 2024-02-10T09:30:19.522 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.520+0000 7f570575c700 1 -- 172.21.15.158:0/1132249194 wait complete. 2024-02-10T09:30:19.523 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f570575c700 1 Processor -- start 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f570575c700 1 -- start start 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f570575c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f570575c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5700106cd0 con 0x7f57000f6510 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f56feffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f56feffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44924/0 (socket says 172.21.15.158:44924) 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.521+0000 7f56feffd700 1 -- 172.21.15.158:0/1130061813 learned_addr learned my addr 172.21.15.158:0/1130061813 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f56feffd700 1 -- 172.21.15.158:0/1130061813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f4005040 con 0x7f57000f6510 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f56feffd700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57001067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f56f4005010 tx=0x7f56f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f4009320 con 0x7f57000f6510 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56f4013380 con 0x7f57000f6510 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700106ed0 con 0x7f57000f6510 2024-02-10T09:30:19.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f4014e50 con 0x7f57000f6510 2024-02-10T09:30:19.525 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.522+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5700107360 con 0x7f57000f6510 2024-02-10T09:30:19.525 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.523+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56f40095f0 con 0x7f57000f6510 2024-02-10T09:30:19.526 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.523+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5700045ad0 con 0x7f57000f6510 2024-02-10T09:30:19.526 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.523+0000 7f56e77fe700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56e802ff10 0x7f56e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:19.526 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.523+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56f4040fa0 con 0x7f57000f6510 2024-02-10T09:30:19.526 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.523+0000 7f56fe7fc700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56e802ff10 0x7f56e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:19.526 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.524+0000 7f56fe7fc700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56e802ff10 0x7f56e80323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f56f00096a0 tx=0x7f56f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:19.529 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.528+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56f4014420 con 0x7f57000f6510 2024-02-10T09:30:19.735 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.733+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5700108210 con 0x7f57000f6510 2024-02-10T09:30:19.735 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.735+0000 7f56e77fe700 1 -- 172.21.15.158:0/1130061813 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56f4010230 con 0x7f57000f6510 2024-02-10T09:30:19.736 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:19.736 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:19.737 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56e802ff10 msgr2=0x7f56e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56e802ff10 0x7f56e80323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f56f00096a0 tx=0x7f56f0006b40).stop 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 msgr2=0x7f57001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57001067e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f56f4005010 tx=0x7f56f400b040).stop 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 shutdown_connections 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f56e802ff10 0x7f56e80323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 --2- 172.21.15.158:0/1130061813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57000f6510 0x7f57001067e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 >> 172.21.15.158:0/1130061813 conn(0x7f57000f1c70 msgr2=0x7f57000f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 shutdown_connections 2024-02-10T09:30:19.738 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:19.736+0000 7f570575c700 1 -- 172.21.15.158:0/1130061813 wait complete. 2024-02-10T09:30:19.739 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:18.111222+0000 mgr.smithi083.dowdym (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:19.735487+0000 mon.smithi083 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.158:0/1130061813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:21.464 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:21.464 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:22.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:20.111640+0000 mgr.smithi083.dowdym (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:23.314 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1994789188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 msgr2=0x7f8cd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 --2- 172.21.15.158:0/1994789188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd80f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f8cc8009fb0 tx=0x7f8cc8009b30).stop 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1994789188 shutdown_connections 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 --2- 172.21.15.158:0/1994789188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd80f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1994789188 >> 172.21.15.158:0/1994789188 conn(0x7f8cd80f0b70 msgr2=0x7f8cd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1994789188 shutdown_connections 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.312+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1994789188 wait complete. 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cdc96b700 1 Processor -- start 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cdc96b700 1 -- start start 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cdc96b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cdc96b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd8106cc0 con 0x7f8cd80f5410 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cd659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:23.315 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cd659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44940/0 (socket says 172.21.15.158:44940) 2024-02-10T09:30:23.316 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.313+0000 7f8cd659c700 1 -- 172.21.15.158:0/1189395603 learned_addr learned my addr 172.21.15.158:0/1189395603 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:23.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cd659c700 1 -- 172.21.15.158:0/1189395603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8005040 con 0x7f8cd80f5410 2024-02-10T09:30:23.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cd659c700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd81067d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8cc8005010 tx=0x7f8cc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:23.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc8009320 con 0x7f8cd80f5410 2024-02-10T09:30:23.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd8106ec0 con 0x7f8cd80f5410 2024-02-10T09:30:23.317 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8cc8013380 con 0x7f8cd80f5410 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc8014e50 con 0x7f8cd80f5410 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.314+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd8107350 con 0x7f8cd80f5410 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.315+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cc80095f0 con 0x7f8cd80f5410 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.315+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd8045ad0 con 0x7f8cd80f5410 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.315+0000 7f8cceffd700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8cc402ff10 0x7f8cc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.315+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cc8040e90 con 0x7f8cd80f5410 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.315+0000 7f8cd5d9b700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8cc402ff10 0x7f8cc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:23.318 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.315+0000 7f8cd5d9b700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8cc402ff10 0x7f8cc40323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8cc00096a0 tx=0x7f8cc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:23.321 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.319+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cc8014420 con 0x7f8cd80f5410 2024-02-10T09:30:23.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.523+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cd8108200 con 0x7f8cd80f5410 2024-02-10T09:30:23.524 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.523+0000 7f8cceffd700 1 -- 172.21.15.158:0/1189395603 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cc8010230 con 0x7f8cd80f5410 2024-02-10T09:30:23.525 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:23.525 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:23.526 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8cc402ff10 msgr2=0x7f8cc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8cc402ff10 0x7f8cc40323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8cc00096a0 tx=0x7f8cc0006b40).stop 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 msgr2=0x7f8cd81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd81067d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f8cc8005010 tx=0x7f8cc800b040).stop 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 shutdown_connections 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8cc402ff10 0x7f8cc40323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 --2- 172.21.15.158:0/1189395603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8cd80f5410 0x7f8cd81067d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 >> 172.21.15.158:0/1189395603 conn(0x7f8cd80f0b70 msgr2=0x7f8cd805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 shutdown_connections 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:23.525+0000 7f8cdc96b700 1 -- 172.21.15.158:0/1189395603 wait complete. 2024-02-10T09:30:23.527 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:24.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:22.111898+0000 mgr.smithi083.dowdym (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:24.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:23.524789+0000 mon.smithi083 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.158:0/1189395603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:25.173 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:25.173 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:26.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:24.112238+0000 mgr.smithi083.dowdym (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 -- 172.21.15.158:0/416472481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 msgr2=0x7f1b3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 --2- 172.21.15.158:0/416472481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c0f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f1b38009fb0 tx=0x7f1b38009b30).stop 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 -- 172.21.15.158:0/416472481 shutdown_connections 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 --2- 172.21.15.158:0/416472481 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c0f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 -- 172.21.15.158:0/416472481 >> 172.21.15.158:0/416472481 conn(0x7f1b3c0f0b70 msgr2=0x7f1b3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 -- 172.21.15.158:0/416472481 shutdown_connections 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 -- 172.21.15.158:0/416472481 wait complete. 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 Processor -- start 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.140+0000 7f1b444de700 1 -- start start 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b444de700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:27.143 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b444de700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c060240 con 0x7f1b3c0f5410 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b4227a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b4227a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:44966/0 (socket says 172.21.15.158:44966) 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b4227a700 1 -- 172.21.15.158:0/2938832803 learned_addr learned my addr 172.21.15.158:0/2938832803 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b4227a700 1 -- 172.21.15.158:0/2938832803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b38005040 con 0x7f1b3c0f5410 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b4227a700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1094e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f1b38005010 tx=0x7f1b3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.141+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b38009320 con 0x7f1b3c0f5410 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c060440 con 0x7f1b3c0f5410 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b38013380 con 0x7f1b3c0f5410 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b38014e50 con 0x7f1b3c0f5410 2024-02-10T09:30:27.144 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c109b90 con 0x7f1b3c0f5410 2024-02-10T09:30:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b380095f0 con 0x7f1b3c0f5410 2024-02-10T09:30:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c045ad0 con 0x7f1b3c0f5410 2024-02-10T09:30:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b32ffd700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b2802ff10 0x7f1b280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.142+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b38040ed0 con 0x7f1b3c0f5410 2024-02-10T09:30:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.143+0000 7f1b41a79700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b2802ff10 0x7f1b280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:27.145 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.143+0000 7f1b41a79700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b2802ff10 0x7f1b280323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1b2c0096a0 tx=0x7f1b2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:27.148 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.148+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b38014420 con 0x7f1b3c0f5410 2024-02-10T09:30:27.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.353+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b3c10a3b0 con 0x7f1b3c0f5410 2024-02-10T09:30:27.354 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.353+0000 7f1b32ffd700 1 -- 172.21.15.158:0/2938832803 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b38010230 con 0x7f1b3c0f5410 2024-02-10T09:30:27.354 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:27.355 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:27.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b2802ff10 msgr2=0x7f1b280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:27.356 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b2802ff10 0x7f1b280323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1b2c0096a0 tx=0x7f1b2c006b40).stop 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 msgr2=0x7f1b3c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1094e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f1b38005010 tx=0x7f1b3800b040).stop 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 shutdown_connections 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f1b2802ff10 0x7f1b280323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 --2- 172.21.15.158:0/2938832803 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b3c0f5410 0x7f1b3c1094e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 >> 172.21.15.158:0/2938832803 conn(0x7f1b3c0f0b70 msgr2=0x7f1b3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 shutdown_connections 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:27.355+0000 7f1b444de700 1 -- 172.21.15.158:0/2938832803 wait complete. 2024-02-10T09:30:27.357 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:28.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:26.112620+0000 mgr.smithi083.dowdym (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:28.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:27.354685+0000 mon.smithi083 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.158:0/2938832803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:29.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:29.033 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:30.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:28.112907+0000 mgr.smithi083.dowdym (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 -- 172.21.15.158:0/150215398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 msgr2=0x7fcc5c0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 --2- 172.21.15.158:0/150215398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c0fb550 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fcc4c009fb0 tx=0x7fcc4c009b30).stop 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 -- 172.21.15.158:0/150215398 shutdown_connections 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 --2- 172.21.15.158:0/150215398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c0fb550 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 -- 172.21.15.158:0/150215398 >> 172.21.15.158:0/150215398 conn(0x7fcc5c0f0b70 msgr2=0x7fcc5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 -- 172.21.15.158:0/150215398 shutdown_connections 2024-02-10T09:30:30.935 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.934+0000 7fcc61c93700 1 -- 172.21.15.158:0/150215398 wait complete. 2024-02-10T09:30:30.936 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc61c93700 1 Processor -- start 2024-02-10T09:30:30.936 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc61c93700 1 -- start start 2024-02-10T09:30:30.937 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc61c93700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c108960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:30.937 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc61c93700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c108e50 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc5b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c108960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc5b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c108960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:48356/0 (socket says 172.21.15.158:48356) 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.935+0000 7fcc5b7fe700 1 -- 172.21.15.158:0/1748748237 learned_addr learned my addr 172.21.15.158:0/1748748237 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc5b7fe700 1 -- 172.21.15.158:0/1748748237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c005040 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc5b7fe700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c108960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fcc4c0089f0 tx=0x7fcc4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc4c009320 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c109050 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc4c013380 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc4c014e20 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.936+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c1094e0 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.937+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc4c0095f0 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.937+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c045ad0 con 0x7fcc5c060220 2024-02-10T09:30:30.938 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.937+0000 7fcc43fff700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc4402ff60 0x7fcc44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:30.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.937+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc4c040f50 con 0x7fcc5c060220 2024-02-10T09:30:30.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.937+0000 7fcc5affd700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc4402ff60 0x7fcc44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:30.939 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.938+0000 7fcc5affd700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc4402ff60 0x7fcc44032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:30.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:30.942+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc4c014420 con 0x7fcc5c060220 2024-02-10T09:30:31.146 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.145+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc5c10a390 con 0x7fcc5c060220 2024-02-10T09:30:31.146 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.146+0000 7fcc43fff700 1 -- 172.21.15.158:0/1748748237 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcc4c01d980 con 0x7fcc5c060220 2024-02-10T09:30:31.147 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:31.147 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:31.148 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc4402ff60 msgr2=0x7fcc44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:31.148 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc4402ff60 0x7fcc44032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).stop 2024-02-10T09:30:31.148 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 msgr2=0x7fcc5c108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c108960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fcc4c0089f0 tx=0x7fcc4c00b040).stop 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 shutdown_connections 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcc4402ff60 0x7fcc44032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 --2- 172.21.15.158:0/1748748237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc5c060220 0x7fcc5c108960 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 >> 172.21.15.158:0/1748748237 conn(0x7fcc5c0f0b70 msgr2=0x7fcc5c0f15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 shutdown_connections 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:31.148+0000 7fcc61c93700 1 -- 172.21.15.158:0/1748748237 wait complete. 2024-02-10T09:30:31.149 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:32.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:30.113230+0000 mgr.smithi083.dowdym (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:32.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:31.146521+0000 mon.smithi083 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.158:0/1748748237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:32.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:32.718 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:34.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:32.113507+0000 mgr.smithi083.dowdym (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:34.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 -- 172.21.15.158:0/3946740615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 msgr2=0x7fed800f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:34.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 --2- 172.21.15.158:0/3946740615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed800f5d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fed74009fb0 tx=0x7fed74009b30).stop 2024-02-10T09:30:34.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 -- 172.21.15.158:0/3946740615 shutdown_connections 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 --2- 172.21.15.158:0/3946740615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed800f5d70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 -- 172.21.15.158:0/3946740615 >> 172.21.15.158:0/3946740615 conn(0x7fed800f0260 msgr2=0x7fed800f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 -- 172.21.15.158:0/3946740615 shutdown_connections 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.811+0000 7fed86799700 1 -- 172.21.15.158:0/3946740615 wait complete. 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed86799700 1 Processor -- start 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed86799700 1 -- start start 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed86799700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed801056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed86799700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80105bd0 con 0x7fed800f5950 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed7ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed801056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed7ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed801056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:48364/0 (socket says 172.21.15.158:48364) 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.812+0000 7fed7ffff700 1 -- 172.21.15.158:0/2040184033 learned_addr learned my addr 172.21.15.158:0/2040184033 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed7ffff700 1 -- 172.21.15.158:0/2040184033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed74005040 con 0x7fed800f5950 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed7ffff700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed801056e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fed74005010 tx=0x7fed7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:34.814 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed74009320 con 0x7fed800f5950 2024-02-10T09:30:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80105dd0 con 0x7fed800f5950 2024-02-10T09:30:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed74013380 con 0x7fed800f5950 2024-02-10T09:30:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed80106260 con 0x7fed800f5950 2024-02-10T09:30:34.815 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.813+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed74014e50 con 0x7fed800f5950 2024-02-10T09:30:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.814+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed740263a0 con 0x7fed800f5950 2024-02-10T09:30:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.814+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed80045ad0 con 0x7fed800f5950 2024-02-10T09:30:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.814+0000 7fed7cff9700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fed6802ff10 0x7fed680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.814+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed74040fb0 con 0x7fed800f5950 2024-02-10T09:30:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.814+0000 7fed7f7fe700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fed6802ff10 0x7fed680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:34.816 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.814+0000 7fed7f7fe700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fed6802ff10 0x7fed680323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fed700096a0 tx=0x7fed70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:34.819 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:34.818+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed74012920 con 0x7fed800f5950 2024-02-10T09:30:35.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.027+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed801070b0 con 0x7fed800f5950 2024-02-10T09:30:35.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.028+0000 7fed7cff9700 1 -- 172.21.15.158:0/2040184033 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed7400f340 con 0x7fed800f5950 2024-02-10T09:30:35.029 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:35.029 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:35.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fed6802ff10 msgr2=0x7fed680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fed6802ff10 0x7fed680323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fed700096a0 tx=0x7fed70006b40).stop 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 msgr2=0x7fed801056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed801056e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fed74005010 tx=0x7fed7400b040).stop 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 shutdown_connections 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fed6802ff10 0x7fed680323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 --2- 172.21.15.158:0/2040184033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed800f5950 0x7fed801056e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 >> 172.21.15.158:0/2040184033 conn(0x7fed800f0260 msgr2=0x7fed800f0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 shutdown_connections 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:35.030+0000 7fed86799700 1 -- 172.21.15.158:0/2040184033 wait complete. 2024-02-10T09:30:35.031 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:36.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:34.113834+0000 mgr.smithi083.dowdym (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:36.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:35.028720+0000 mon.smithi083 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.158:0/2040184033' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:36.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:36.674 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:38.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:36.114235+0000 mgr.smithi083.dowdym (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:38.645 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 -- 172.21.15.158:0/1088565159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 msgr2=0x7faf200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 --2- 172.21.15.158:0/1088565159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf200f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7faf08009fb0 tx=0x7faf08009b30).stop 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 -- 172.21.15.158:0/1088565159 shutdown_connections 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 --2- 172.21.15.158:0/1088565159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf200f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 -- 172.21.15.158:0/1088565159 >> 172.21.15.158:0/1088565159 conn(0x7faf200f1c70 msgr2=0x7faf200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 -- 172.21.15.158:0/1088565159 shutdown_connections 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.643+0000 7faf26458700 1 -- 172.21.15.158:0/1088565159 wait complete. 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf26458700 1 Processor -- start 2024-02-10T09:30:38.646 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf26458700 1 -- start start 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf26458700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf26458700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf20106cd0 con 0x7faf200f6510 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf1ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:48378/0 (socket says 172.21.15.158:48378) 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.644+0000 7faf1ffff700 1 -- 172.21.15.158:0/253420391 learned_addr learned my addr 172.21.15.158:0/253420391 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf1ffff700 1 -- 172.21.15.158:0/253420391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08005040 con 0x7faf200f6510 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf1ffff700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf201067e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7faf08005010 tx=0x7faf0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf08009320 con 0x7faf200f6510 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf20106ed0 con 0x7faf200f6510 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf08013380 con 0x7faf200f6510 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf08014e50 con 0x7faf200f6510 2024-02-10T09:30:38.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.645+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf20107360 con 0x7faf200f6510 2024-02-10T09:30:38.648 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.646+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf080263a0 con 0x7faf200f6510 2024-02-10T09:30:38.648 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.646+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf20045ad0 con 0x7faf200f6510 2024-02-10T09:30:38.648 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.646+0000 7faf1cff9700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faf0c02ff10 0x7faf0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:38.648 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.646+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf080412b0 con 0x7faf200f6510 2024-02-10T09:30:38.648 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.646+0000 7faf1f7fe700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faf0c02ff10 0x7faf0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:38.649 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.647+0000 7faf1f7fe700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faf0c02ff10 0x7faf0c0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7faf100096a0 tx=0x7faf10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:38.651 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.650+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf08014420 con 0x7faf200f6510 2024-02-10T09:30:38.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.855+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf20108210 con 0x7faf200f6510 2024-02-10T09:30:38.856 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.856+0000 7faf1cff9700 1 -- 172.21.15.158:0/253420391 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf08012330 con 0x7faf200f6510 2024-02-10T09:30:38.856 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:38.856 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faf0c02ff10 msgr2=0x7faf0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faf0c02ff10 0x7faf0c0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7faf100096a0 tx=0x7faf10006b40).stop 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 msgr2=0x7faf201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf201067e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7faf08005010 tx=0x7faf0800b040).stop 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 shutdown_connections 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7faf0c02ff10 0x7faf0c0323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 --2- 172.21.15.158:0/253420391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf200f6510 0x7faf201067e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 >> 172.21.15.158:0/253420391 conn(0x7faf200f1c70 msgr2=0x7faf200f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 shutdown_connections 2024-02-10T09:30:38.858 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:38.857+0000 7faf26458700 1 -- 172.21.15.158:0/253420391 wait complete. 2024-02-10T09:30:38.859 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:39.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:38 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:38.855946+0000 mon.smithi083 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.158:0/253420391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:40.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:38.114620+0000 mgr.smithi083.dowdym (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:40.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:40.546 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:40.114956+0000 mgr.smithi083.dowdym (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:42.465 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 -- 172.21.15.158:0/4257158564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 msgr2=0x7f3d4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:42.465 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 --2- 172.21.15.158:0/4257158564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f3d48009fb0 tx=0x7f3d48009b30).stop 2024-02-10T09:30:42.465 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 -- 172.21.15.158:0/4257158564 shutdown_connections 2024-02-10T09:30:42.465 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 --2- 172.21.15.158:0/4257158564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:42.465 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 -- 172.21.15.158:0/4257158564 >> 172.21.15.158:0/4257158564 conn(0x7f3d4c0f0b70 msgr2=0x7f3d4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 -- 172.21.15.158:0/4257158564 shutdown_connections 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.457+0000 7f3d534b2700 1 -- 172.21.15.158:0/4257158564 wait complete. 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d534b2700 1 Processor -- start 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d534b2700 1 -- start start 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d534b2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d534b2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c105bd0 con 0x7f3d4c0f5410 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d5124e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d5124e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52176/0 (socket says 172.21.15.158:52176) 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d5124e700 1 -- 172.21.15.158:0/637544586 learned_addr learned my addr 172.21.15.158:0/637544586 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.458+0000 7f3d5124e700 1 -- 172.21.15.158:0/637544586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d48005040 con 0x7f3d4c0f5410 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.459+0000 7f3d5124e700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c1056e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f3d48005010 tx=0x7f3d4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.459+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d48009320 con 0x7f3d4c0f5410 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.459+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c105dd0 con 0x7f3d4c0f5410 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.459+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d48013380 con 0x7f3d4c0f5410 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.459+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d48014e50 con 0x7f3d4c0f5410 2024-02-10T09:30:42.466 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.459+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c106260 con 0x7f3d4c0f5410 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.460+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c045ad0 con 0x7f3d4c0f5410 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.460+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d480095f0 con 0x7f3d4c0f5410 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.460+0000 7f3d41ffb700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3d3802ff10 0x7f3d380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.460+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d48040e90 con 0x7f3d4c0f5410 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.460+0000 7f3d50a4d700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3d3802ff10 0x7f3d380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.461+0000 7f3d50a4d700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3d3802ff10 0x7f3d380323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3d3c0096a0 tx=0x7f3d3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:42.467 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.464+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d48014420 con 0x7f3d4c0f5410 2024-02-10T09:30:42.670 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.670+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d4c107110 con 0x7f3d4c0f5410 2024-02-10T09:30:42.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.671+0000 7f3d41ffb700 1 -- 172.21.15.158:0/637544586 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d48010230 con 0x7f3d4c0f5410 2024-02-10T09:30:42.671 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:42.671 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3d3802ff10 msgr2=0x7f3d380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3d3802ff10 0x7f3d380323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f3d3c0096a0 tx=0x7f3d3c006b40).stop 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 msgr2=0x7f3d4c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c1056e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f3d48005010 tx=0x7f3d4800b040).stop 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 shutdown_connections 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3d3802ff10 0x7f3d380323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 --2- 172.21.15.158:0/637544586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d4c0f5410 0x7f3d4c1056e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 >> 172.21.15.158:0/637544586 conn(0x7f3d4c0f0b70 msgr2=0x7f3d4c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:42.673 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 shutdown_connections 2024-02-10T09:30:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:42.672+0000 7f3d534b2700 1 -- 172.21.15.158:0/637544586 wait complete. 2024-02-10T09:30:42.674 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:43.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:42 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:42.671132+0000 mon.smithi083 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.158:0/637544586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:44.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:42.115274+0000 mgr.smithi083.dowdym (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:44.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:44.446 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:46.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:44.115647+0000 mgr.smithi083.dowdym (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 -- 172.21.15.158:0/4012342386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 msgr2=0x7f01400f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 --2- 172.21.15.158:0/4012342386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01400f66c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f0130009fb0 tx=0x7f0130009b30).stop 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 -- 172.21.15.158:0/4012342386 shutdown_connections 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 --2- 172.21.15.158:0/4012342386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01400f66c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 -- 172.21.15.158:0/4012342386 >> 172.21.15.158:0/4012342386 conn(0x7f01400f1aa0 msgr2=0x7f01400f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 -- 172.21.15.158:0/4012342386 shutdown_connections 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 -- 172.21.15.158:0/4012342386 wait complete. 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 Processor -- start 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.380+0000 7f013f59e700 1 -- start start 2024-02-10T09:30:46.382 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013f59e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01401065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013f59e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0140106ac0 con 0x7f01400f62a0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01401065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013e59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01401065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52186/0 (socket says 172.21.15.158:52186) 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013e59c700 1 -- 172.21.15.158:0/564905272 learned_addr learned my addr 172.21.15.158:0/564905272 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013e59c700 1 -- 172.21.15.158:0/564905272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0130005040 con 0x7f01400f62a0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013e59c700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01401065d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f01300089f0 tx=0x7f013000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0130009320 con 0x7f01400f62a0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0140106cc0 con 0x7f01400f62a0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0130013380 con 0x7f01400f62a0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.381+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0130014e20 con 0x7f01400f62a0 2024-02-10T09:30:46.383 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.382+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0140107150 con 0x7f01400f62a0 2024-02-10T09:30:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.382+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01300095f0 con 0x7f01400f62a0 2024-02-10T09:30:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.382+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0140045ad0 con 0x7f01400f62a0 2024-02-10T09:30:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.382+0000 7f012effd700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f012802ff10 0x7f01280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:46.384 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.382+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0130040f50 con 0x7f01400f62a0 2024-02-10T09:30:46.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.383+0000 7f013dd9b700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f012802ff10 0x7f01280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:46.385 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.383+0000 7f013dd9b700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f012802ff10 0x7f01280323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f01340096a0 tx=0x7f0134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:46.387 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.387+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0130014420 con 0x7f01400f62a0 2024-02-10T09:30:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.593+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0140108000 con 0x7f01400f62a0 2024-02-10T09:30:46.594 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.594+0000 7f012effd700 1 -- 172.21.15.158:0/564905272 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f013001d980 con 0x7f01400f62a0 2024-02-10T09:30:46.594 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:46.594 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f012802ff10 msgr2=0x7f01280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f012802ff10 0x7f01280323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f01340096a0 tx=0x7f0134006b40).stop 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 msgr2=0x7f01401065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01401065d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f01300089f0 tx=0x7f013000b040).stop 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 shutdown_connections 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f012802ff10 0x7f01280323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 --2- 172.21.15.158:0/564905272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01400f62a0 0x7f01401065d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.595+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 >> 172.21.15.158:0/564905272 conn(0x7f01400f1aa0 msgr2=0x7f01400f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:46.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.596+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 shutdown_connections 2024-02-10T09:30:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:46.596+0000 7f013f59e700 1 -- 172.21.15.158:0/564905272 wait complete. 2024-02-10T09:30:46.597 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:47.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:46 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:46.594461+0000 mon.smithi083 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.158:0/564905272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:46.115970+0000 mgr.smithi083.dowdym (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:48.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:48.298 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:50.268 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 -- 172.21.15.158:0/3413818355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f40f51c0 msgr2=0x7f83f40f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 --2- 172.21.15.158:0/3413818355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f40f51c0 0x7f83f40f55e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f83e8009fb0 tx=0x7f83e8009b30).stop 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 -- 172.21.15.158:0/3413818355 shutdown_connections 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 --2- 172.21.15.158:0/3413818355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f40f51c0 0x7f83f40f55e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 -- 172.21.15.158:0/3413818355 >> 172.21.15.158:0/3413818355 conn(0x7f83f40f0920 msgr2=0x7f83f40f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 -- 172.21.15.158:0/3413818355 shutdown_connections 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.214+0000 7f83f933b700 1 -- 172.21.15.158:0/3413818355 wait complete. 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f933b700 1 Processor -- start 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f933b700 1 -- start start 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f933b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 0x7f83f4106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f933b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f4107090 con 0x7f83f4106780 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 0x7f83f4106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:50.269 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 0x7f83f4106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46478/0 (socket says 172.21.15.158:46478) 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.215+0000 7f83f2ffd700 1 -- 172.21.15.158:0/2099872773 learned_addr learned my addr 172.21.15.158:0/2099872773 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83f2ffd700 1 -- 172.21.15.158:0/2099872773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8005040 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83f2ffd700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 0x7f83f4106ba0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f83f40f62c0 tx=0x7f83e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83e8013380 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f4107290 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83e8013790 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83e801ee70 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.216+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f4107de0 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.217+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83e8025060 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.217+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f4059100 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.217+0000 7f83db7fe700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83dc02ff60 0x7f83dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.217+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83e8040dc0 con 0x7f83f4106780 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.217+0000 7f83f27fc700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83dc02ff60 0x7f83dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.218+0000 7f83f27fc700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83dc02ff60 0x7f83dc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f83e40096a0 tx=0x7f83e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:50.270 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.221+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83e801c950 con 0x7f83f4106780 2024-02-10T09:30:50.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:48.116315+0000 mgr.smithi083.dowdym (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:50.430 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.430+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83f4108530 con 0x7f83f4106780 2024-02-10T09:30:50.431 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.430+0000 7f83db7fe700 1 -- 172.21.15.158:0/2099872773 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83e802d2d0 con 0x7f83f4106780 2024-02-10T09:30:50.431 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:50.431 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83dc02ff60 msgr2=0x7f83dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83dc02ff60 0x7f83dc032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f83e40096a0 tx=0x7f83e4006b40).stop 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 msgr2=0x7f83f4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 0x7f83f4106ba0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f83f40f62c0 tx=0x7f83e8009b30).stop 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 shutdown_connections 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f83dc02ff60 0x7f83dc032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 --2- 172.21.15.158:0/2099872773 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83f4106780 0x7f83f4106ba0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 >> 172.21.15.158:0/2099872773 conn(0x7f83f40f0920 msgr2=0x7f83f405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:50.433 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 shutdown_connections 2024-02-10T09:30:50.434 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:50.432+0000 7f83f933b700 1 -- 172.21.15.158:0/2099872773 wait complete. 2024-02-10T09:30:50.434 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:51.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:50 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:50.430933+0000 mon.smithi083 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.158:0/2099872773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:52.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:52.123 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:52.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:50.116687+0000 mgr.smithi083.dowdym (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/879977245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 msgr2=0x7f2d540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 --2- 172.21.15.158:0/879977245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d540f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f2d44009fb0 tx=0x7f2d44009b30).stop 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/879977245 shutdown_connections 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 --2- 172.21.15.158:0/879977245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d540f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/879977245 >> 172.21.15.158:0/879977245 conn(0x7f2d540f0b70 msgr2=0x7f2d540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/879977245 shutdown_connections 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/879977245 wait complete. 2024-02-10T09:30:54.124 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.122+0000 7f2d5b6f0700 1 Processor -- start 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5b6f0700 1 -- start start 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5b6f0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d541067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5b6f0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54106cd0 con 0x7f2d540f5410 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5948c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d541067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5948c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d541067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46502/0 (socket says 172.21.15.158:46502) 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5948c700 1 -- 172.21.15.158:0/2001615384 learned_addr learned my addr 172.21.15.158:0/2001615384 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5948c700 1 -- 172.21.15.158:0/2001615384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44005040 con 0x7f2d540f5410 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.123+0000 7f2d5948c700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d541067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f2d44005010 tx=0x7f2d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.124+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d44009320 con 0x7f2d540f5410 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.124+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d54106ed0 con 0x7f2d540f5410 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.124+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d44013380 con 0x7f2d540f5410 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.124+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d44014e50 con 0x7f2d540f5410 2024-02-10T09:30:54.125 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.124+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d54107360 con 0x7f2d540f5410 2024-02-10T09:30:54.126 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.125+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d54045ad0 con 0x7f2d540f5410 2024-02-10T09:30:54.126 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.125+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d440095f0 con 0x7f2d540f5410 2024-02-10T09:30:54.126 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.125+0000 7f2d49ffb700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2d4002ff10 0x7f2d400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:54.126 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.125+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d44040e90 con 0x7f2d540f5410 2024-02-10T09:30:54.126 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.125+0000 7f2d58c8b700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2d4002ff10 0x7f2d400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:54.127 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.125+0000 7f2d58c8b700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2d4002ff10 0x7f2d400323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f2d500096a0 tx=0x7f2d50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:54.129 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.129+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d44014420 con 0x7f2d540f5410 2024-02-10T09:30:54.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:52.117037+0000 mgr.smithi083.dowdym (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:54.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.337+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d540fafa0 con 0x7f2d540f5410 2024-02-10T09:30:54.338 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.338+0000 7f2d49ffb700 1 -- 172.21.15.158:0/2001615384 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d44010230 con 0x7f2d540f5410 2024-02-10T09:30:54.338 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:54.338 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:54.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2d4002ff10 msgr2=0x7f2d400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:54.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2d4002ff10 0x7f2d400323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f2d500096a0 tx=0x7f2d50006b40).stop 2024-02-10T09:30:54.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 msgr2=0x7f2d541067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:54.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d541067e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f2d44005010 tx=0x7f2d4400b040).stop 2024-02-10T09:30:54.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 shutdown_connections 2024-02-10T09:30:54.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2d4002ff10 0x7f2d400323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:54.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 --2- 172.21.15.158:0/2001615384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d540f5410 0x7f2d541067e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:54.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 >> 172.21.15.158:0/2001615384 conn(0x7f2d540f0b70 msgr2=0x7f2d5405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:54.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 shutdown_connections 2024-02-10T09:30:54.341 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:54.340+0000 7f2d5b6f0700 1 -- 172.21.15.158:0/2001615384 wait complete. 2024-02-10T09:30:54.341 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:55.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:54 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:54.338137+0000 mon.smithi083 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.158:0/2001615384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:56.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:56.109 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:30:56.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:54.117405+0000 mgr.smithi083.dowdym (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 -- 172.21.15.158:0/1179855879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 msgr2=0x7f10580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 --2- 172.21.15.158:0/1179855879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f10580f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f1048009fb0 tx=0x7f1048009b30).stop 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 -- 172.21.15.158:0/1179855879 shutdown_connections 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 --2- 172.21.15.158:0/1179855879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f10580f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 -- 172.21.15.158:0/1179855879 >> 172.21.15.158:0/1179855879 conn(0x7f10580f0b70 msgr2=0x7f10580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 -- 172.21.15.158:0/1179855879 shutdown_connections 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.074+0000 7f105f841700 1 -- 172.21.15.158:0/1179855879 wait complete. 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105f841700 1 Processor -- start 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105f841700 1 -- start start 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105f841700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f1058109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:58.080 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105f841700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058060220 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105d5dd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f1058109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105d5dd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f1058109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:46520/0 (socket says 172.21.15.158:46520) 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105d5dd700 1 -- 172.21.15.158:0/609543034 learned_addr learned my addr 172.21.15.158:0/609543034 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105d5dd700 1 -- 172.21.15.158:0/609543034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1048005040 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.075+0000 7f105d5dd700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f1058109500 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f1048005010 tx=0x7f104800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.076+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1048009320 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.076+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1058060420 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.076+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1048013380 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.076+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1048014e50 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.076+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1058109bb0 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.077+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10480095f0 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.077+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058059100 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.077+0000 7f104dffb700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f104402ff10 0x7f10440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.077+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1048040f80 con 0x7f10580f5410 2024-02-10T09:30:58.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.077+0000 7f105cddc700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f104402ff10 0x7f10440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:30:58.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.077+0000 7f105cddc700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f104402ff10 0x7f10440323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f10540096a0 tx=0x7f1054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:30:58.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.081+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1048014420 con 0x7f10580f5410 2024-02-10T09:30:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:56.117689+0000 mgr.smithi083.dowdym (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:30:58.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.286+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1058045ad0 con 0x7f10580f5410 2024-02-10T09:30:58.287 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.286+0000 7f104dffb700 1 -- 172.21.15.158:0/609543034 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1048010230 con 0x7f10580f5410 2024-02-10T09:30:58.288 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:30:58.288 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:30:58.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f104402ff10 msgr2=0x7f10440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:58.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f104402ff10 0x7f10440323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f10540096a0 tx=0x7f1054006b40).stop 2024-02-10T09:30:58.289 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 msgr2=0x7f1058109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f1058109500 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f1048005010 tx=0x7f104800b040).stop 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 shutdown_connections 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f104402ff10 0x7f10440323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 --2- 172.21.15.158:0/609543034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10580f5410 0x7f1058109500 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 >> 172.21.15.158:0/609543034 conn(0x7f10580f0b70 msgr2=0x7f10580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 shutdown_connections 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:30:58.288+0000 7f105f841700 1 -- 172.21.15.158:0/609543034 wait complete. 2024-02-10T09:30:58.290 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:30:59.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:58 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:30:58.287844+0000 mon.smithi083 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.158:0/609543034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:30:59.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:30:59.996 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:30:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:30:58.118041+0000 mgr.smithi083.dowdym (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 -- 172.21.15.158:0/3443107578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 msgr2=0x7f5e3c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 --2- 172.21.15.158:0/3443107578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c0f55c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f5e24009fb0 tx=0x7f5e24009b30).stop 2024-02-10T09:31:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 -- 172.21.15.158:0/3443107578 shutdown_connections 2024-02-10T09:31:01.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 --2- 172.21.15.158:0/3443107578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c0f55c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 -- 172.21.15.158:0/3443107578 >> 172.21.15.158:0/3443107578 conn(0x7f5e3c0f09a0 msgr2=0x7f5e3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 -- 172.21.15.158:0/3443107578 shutdown_connections 2024-02-10T09:31:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.841+0000 7f5e40fec700 1 -- 172.21.15.158:0/3443107578 wait complete. 2024-02-10T09:31:01.844 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.842+0000 7f5e40fec700 1 Processor -- start 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.842+0000 7f5e40fec700 1 -- start start 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.842+0000 7f5e40fec700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.842+0000 7f5e40fec700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1059c0 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e3b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e3b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:42052/0 (socket says 172.21.15.158:42052) 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e3b7fe700 1 -- 172.21.15.158:0/1202665012 learned_addr learned my addr 172.21.15.158:0/1202665012 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e3b7fe700 1 -- 172.21.15.158:0/1202665012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24005040 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.846 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e3b7fe700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c1054d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5e240089f0 tx=0x7f5e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e24009320 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c105bc0 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e24013380 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.843+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e24014e20 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.844+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c106050 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.844+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e240263a0 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.844+0000 7f5e33fff700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e2802ff10 0x7f5e280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.844+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e24040f90 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.845+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c045ad0 con 0x7f5e3c0f51a0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.845+0000 7f5e3affd700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e2802ff10 0x7f5e280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:01.847 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.845+0000 7f5e3affd700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e2802ff10 0x7f5e280323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f5e2c0096a0 tx=0x7f5e2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:01.850 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:01.849+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e24012920 con 0x7f5e3c0f51a0 2024-02-10T09:31:02.055 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.054+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e3c106da0 con 0x7f5e3c0f51a0 2024-02-10T09:31:02.056 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.055+0000 7f5e33fff700 1 -- 172.21.15.158:0/1202665012 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e2402b020 con 0x7f5e3c0f51a0 2024-02-10T09:31:02.057 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:02.057 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:02.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e2802ff10 msgr2=0x7f5e280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:02.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e2802ff10 0x7f5e280323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f5e2c0096a0 tx=0x7f5e2c006b40).stop 2024-02-10T09:31:02.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 msgr2=0x7f5e3c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c1054d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5e240089f0 tx=0x7f5e2400b040).stop 2024-02-10T09:31:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 shutdown_connections 2024-02-10T09:31:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5e2802ff10 0x7f5e280323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 --2- 172.21.15.158:0/1202665012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e3c0f51a0 0x7f5e3c1054d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 >> 172.21.15.158:0/1202665012 conn(0x7f5e3c0f09a0 msgr2=0x7f5e3c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:02.059 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 shutdown_connections 2024-02-10T09:31:02.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:02.056+0000 7f5e40fec700 1 -- 172.21.15.158:0/1202665012 wait complete. 2024-02-10T09:31:02.060 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:02.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:00.118356+0000 mgr.smithi083.dowdym (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:03.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:02 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:02.055949+0000 mon.smithi083 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.158:0/1202665012' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:03.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:03.700 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:04.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:02.118735+0000 mgr.smithi083.dowdym (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:05.748 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 -- 172.21.15.158:0/3247879513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 msgr2=0x7ff4900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 --2- 172.21.15.158:0/3247879513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7ff478009fb0 tx=0x7ff478009b30).stop 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 -- 172.21.15.158:0/3247879513 shutdown_connections 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 --2- 172.21.15.158:0/3247879513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 -- 172.21.15.158:0/3247879513 >> 172.21.15.158:0/3247879513 conn(0x7ff4900f0b70 msgr2=0x7ff4900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 -- 172.21.15.158:0/3247879513 shutdown_connections 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.708+0000 7ff496439700 1 -- 172.21.15.158:0/3247879513 wait complete. 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.709+0000 7ff496439700 1 Processor -- start 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.709+0000 7ff496439700 1 -- start start 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.709+0000 7ff496439700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:05.749 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.709+0000 7ff496439700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900f4ff0 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:42058/0 (socket says 172.21.15.158:42058) 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48ffff700 1 -- 172.21.15.158:0/2920152497 learned_addr learned my addr 172.21.15.158:0/2920152497 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48ffff700 1 -- 172.21.15.158:0/2920152497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478005040 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48ffff700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f6b70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7ff4780089f0 tx=0x7ff47800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff478009320 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4900f5190 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff478013380 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff478014e20 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.710+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4900f5620 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.711+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4780095f0 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.711+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff490059100 con 0x7ff490060280 2024-02-10T09:31:05.750 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.711+0000 7ff48cff9700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff47c02ff10 0x7ff47c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:05.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.711+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff478040f50 con 0x7ff490060280 2024-02-10T09:31:05.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.712+0000 7ff48f7fe700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff47c02ff10 0x7ff47c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:05.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.712+0000 7ff48f7fe700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff47c02ff10 0x7ff47c0323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ff4800096a0 tx=0x7ff480006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:05.751 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.716+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff478014420 con 0x7ff490060280 2024-02-10T09:31:05.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.921+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4900fc090 con 0x7ff490060280 2024-02-10T09:31:05.923 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.921+0000 7ff48cff9700 1 -- 172.21.15.158:0/2920152497 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff47801d980 con 0x7ff490060280 2024-02-10T09:31:05.923 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:05.923 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff47c02ff10 msgr2=0x7ff47c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff47c02ff10 0x7ff47c0323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ff4800096a0 tx=0x7ff480006b40).stop 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 msgr2=0x7ff4900f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f6b70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7ff4780089f0 tx=0x7ff47800b040).stop 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 shutdown_connections 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff47c02ff10 0x7ff47c0323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 --2- 172.21.15.158:0/2920152497 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff490060280 0x7ff4900f6b70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:05.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 >> 172.21.15.158:0/2920152497 conn(0x7ff4900f0b70 msgr2=0x7ff4900f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:05.926 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 shutdown_connections 2024-02-10T09:31:05.926 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:05.923+0000 7ff496439700 1 -- 172.21.15.158:0/2920152497 wait complete. 2024-02-10T09:31:05.926 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:06.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:04.119129+0000 mgr.smithi083.dowdym (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:06.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:05.922723+0000 mon.smithi083 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.158:0/2920152497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:07.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:07.656 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:08.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:06.119501+0000 mgr.smithi083.dowdym (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:09.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 -- 172.21.15.158:0/693579001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 msgr2=0x7f5d4c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:09.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 --2- 172.21.15.158:0/693579001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c0f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f5d40009fb0 tx=0x7f5d40009b30).stop 2024-02-10T09:31:09.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 -- 172.21.15.158:0/693579001 shutdown_connections 2024-02-10T09:31:09.682 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 --2- 172.21.15.158:0/693579001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c0f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 -- 172.21.15.158:0/693579001 >> 172.21.15.158:0/693579001 conn(0x7f5d4c0f1c70 msgr2=0x7f5d4c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 -- 172.21.15.158:0/693579001 shutdown_connections 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.680+0000 7f5d52b1e700 1 -- 172.21.15.158:0/693579001 wait complete. 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d52b1e700 1 Processor -- start 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d52b1e700 1 -- start start 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d52b1e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:09.683 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d52b1e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c106cd0 con 0x7f5d4c0f6510 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d508ba700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d508ba700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53100/0 (socket says 172.21.15.158:53100) 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.681+0000 7f5d508ba700 1 -- 172.21.15.158:0/1637905002 learned_addr learned my addr 172.21.15.158:0/1637905002 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d508ba700 1 -- 172.21.15.158:0/1637905002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d40005040 con 0x7f5d4c0f6510 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d508ba700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c1067e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f5d40005010 tx=0x7f5d4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d40009320 con 0x7f5d4c0f6510 2024-02-10T09:31:09.685 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c106ed0 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d40013380 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d40014e50 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.682+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c107360 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.683+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d400095f0 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.683+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c045ad0 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.683+0000 7f5d497fa700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.683+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d40040e90 con 0x7f5d4c0f6510 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.683+0000 7f5d4bfff700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:09.686 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.684+0000 7f5d4bfff700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5d380096a0 tx=0x7f5d38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:09.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.688+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d40014420 con 0x7f5d4c0f6510 2024-02-10T09:31:09.896 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.895+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d4c108210 con 0x7f5d4c0f6510 2024-02-10T09:31:09.897 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.895+0000 7f5d497fa700 1 -- 172.21.15.158:0/1637905002 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d40010230 con 0x7f5d4c0f6510 2024-02-10T09:31:09.897 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:09.897 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d3c02ff10 msgr2=0x7f5d3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5d380096a0 tx=0x7f5d38006b40).stop 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 msgr2=0x7f5d4c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c1067e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f5d40005010 tx=0x7f5d4000b040).stop 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 shutdown_connections 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:09.899 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 --2- 172.21.15.158:0/1637905002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d4c0f6510 0x7f5d4c1067e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.897+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 >> 172.21.15.158:0/1637905002 conn(0x7f5d4c0f1c70 msgr2=0x7f5d4c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.898+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 shutdown_connections 2024-02-10T09:31:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:09.898+0000 7f5d52b1e700 1 -- 172.21.15.158:0/1637905002 wait complete. 2024-02-10T09:31:09.900 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:10.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:08.119823+0000 mgr.smithi083.dowdym (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:10.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:09.897047+0000 mon.smithi083 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.158:0/1637905002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:11.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:11.704 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:10.120127+0000 mgr.smithi083.dowdym (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.690+0000 7fb1f39c4700 1 -- 172.21.15.158:0/1369067990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec0f5410 msgr2=0x7fb1ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.690+0000 7fb1f39c4700 1 --2- 172.21.15.158:0/1369067990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec0f5410 0x7fb1ec0f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fb1e0009fb0 tx=0x7fb1e0009b30).stop 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.690+0000 7fb1f39c4700 1 -- 172.21.15.158:0/1369067990 shutdown_connections 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.690+0000 7fb1f39c4700 1 --2- 172.21.15.158:0/1369067990 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec0f5410 0x7fb1ec0f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.690+0000 7fb1f39c4700 1 -- 172.21.15.158:0/1369067990 >> 172.21.15.158:0/1369067990 conn(0x7fb1ec0f0b70 msgr2=0x7fb1ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.691+0000 7fb1f39c4700 1 -- 172.21.15.158:0/1369067990 shutdown_connections 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.691+0000 7fb1f39c4700 1 -- 172.21.15.158:0/1369067990 wait complete. 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.691+0000 7fb1f39c4700 1 Processor -- start 2024-02-10T09:31:13.694 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.691+0000 7fb1f39c4700 1 -- start start 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.691+0000 7fb1f39c4700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 0x7fb1ec105c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.691+0000 7fb1f39c4700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ec106110 con 0x7fb1ec105800 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f1760700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 0x7fb1ec105c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f1760700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 0x7fb1ec105c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53112/0 (socket says 172.21.15.158:53112) 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f1760700 1 -- 172.21.15.158:0/2913655781 learned_addr learned my addr 172.21.15.158:0/2913655781 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f1760700 1 -- 172.21.15.158:0/2913655781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e0005040 con 0x7fb1ec105800 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f1760700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 0x7fb1ec105c20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fb1e0009f80 tx=0x7fb1e0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e0013380 con 0x7fb1ec105800 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ec106310 con 0x7fb1ec105800 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1e0013790 con 0x7fb1ec105800 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e0014420 con 0x7fb1ec105800 2024-02-10T09:31:13.695 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.692+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ec106e60 con 0x7fb1ec105800 2024-02-10T09:31:13.696 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.693+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1e0025060 con 0x7fb1ec105800 2024-02-10T09:31:13.696 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.693+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ec059100 con 0x7fb1ec105800 2024-02-10T09:31:13.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.693+0000 7fb1ddffb700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb1d802ff60 0x7fb1d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:13.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.693+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1e0040e10 con 0x7fb1ec105800 2024-02-10T09:31:13.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.694+0000 7fb1f0f5f700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb1d802ff60 0x7fb1d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:13.697 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.694+0000 7fb1f0f5f700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb1d802ff60 0x7fb1d8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb1e80096a0 tx=0x7fb1e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:13.700 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.698+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1e000f280 con 0x7fb1ec105800 2024-02-10T09:31:13.904 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.903+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1ec045ad0 con 0x7fb1ec105800 2024-02-10T09:31:13.905 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.904+0000 7fb1ddffb700 1 -- 172.21.15.158:0/2913655781 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb1e002d2d0 con 0x7fb1ec105800 2024-02-10T09:31:13.905 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:13.905 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:13.907 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb1d802ff60 msgr2=0x7fb1d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:13.907 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb1d802ff60 0x7fb1d8032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fb1e80096a0 tx=0x7fb1e8006b40).stop 2024-02-10T09:31:13.907 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 msgr2=0x7fb1ec105c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:13.907 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 0x7fb1ec105c20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fb1e0009f80 tx=0x7fb1e0009b30).stop 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 shutdown_connections 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb1d802ff60 0x7fb1d8032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 --2- 172.21.15.158:0/2913655781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1ec105800 0x7fb1ec105c20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 >> 172.21.15.158:0/2913655781 conn(0x7fb1ec0f0b70 msgr2=0x7fb1ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 shutdown_connections 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:13.906+0000 7fb1f39c4700 1 -- 172.21.15.158:0/2913655781 wait complete. 2024-02-10T09:31:13.908 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:14.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:12.120412+0000 mgr.smithi083.dowdym (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:14.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:13.905174+0000 mon.smithi083 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.158:0/2913655781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:14.175327+0000 mon.smithi083 (mon.0) 277 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:31:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:14.182736+0000 mon.smithi083 (mon.0) 278 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:31:15.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:15.636 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:14.120682+0000 mgr.smithi083.dowdym (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:17.608 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 -- 172.21.15.158:0/2316040096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 msgr2=0x7f417c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 --2- 172.21.15.158:0/2316040096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c0f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f4170009fb0 tx=0x7f4170009b30).stop 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 -- 172.21.15.158:0/2316040096 shutdown_connections 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 --2- 172.21.15.158:0/2316040096 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c0f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 -- 172.21.15.158:0/2316040096 >> 172.21.15.158:0/2316040096 conn(0x7f417c0f0b70 msgr2=0x7f417c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 -- 172.21.15.158:0/2316040096 shutdown_connections 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.606+0000 7f4181970700 1 -- 172.21.15.158:0/2316040096 wait complete. 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.607+0000 7f4181970700 1 Processor -- start 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.607+0000 7f4181970700 1 -- start start 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.607+0000 7f4181970700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:17.609 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.607+0000 7f4181970700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c060240 con 0x7f417c0f5410 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f417affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f417affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:53138/0 (socket says 172.21.15.158:53138) 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f417affd700 1 -- 172.21.15.158:0/856078182 learned_addr learned my addr 172.21.15.158:0/856078182 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f417affd700 1 -- 172.21.15.158:0/856078182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170005040 con 0x7f417c0f5410 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f417affd700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c1094e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f41700089f0 tx=0x7f417000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4170009320 con 0x7f417c0f5410 2024-02-10T09:31:17.611 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c060440 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4170013380 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417001ed80 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.608+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c109b90 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.609+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f417001e420 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.609+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c045ad0 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.609+0000 7f41637fe700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f416402ff60 0x7f4164032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.609+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4170041110 con 0x7f417c0f5410 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.610+0000 7f417a7fc700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f416402ff60 0x7f4164032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:17.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.610+0000 7f417a7fc700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f416402ff60 0x7f4164032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f416c0096a0 tx=0x7f416c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:17.615 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.614+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f417001cb10 con 0x7f417c0f5410 2024-02-10T09:31:17.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.819+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f417c10a3b0 con 0x7f417c0f5410 2024-02-10T09:31:17.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.820+0000 7f41637fe700 1 -- 172.21.15.158:0/856078182 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4170004490 con 0x7f417c0f5410 2024-02-10T09:31:17.821 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:17.821 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f416402ff60 msgr2=0x7f4164032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:17.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f416402ff60 0x7f4164032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f416c0096a0 tx=0x7f416c006b40).stop 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 msgr2=0x7f417c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c1094e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f41700089f0 tx=0x7f417000b040).stop 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 shutdown_connections 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f416402ff60 0x7f4164032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 --2- 172.21.15.158:0/856078182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f417c0f5410 0x7f417c1094e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 >> 172.21.15.158:0/856078182 conn(0x7f417c0f0b70 msgr2=0x7f417c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 shutdown_connections 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:17.821+0000 7f4181970700 1 -- 172.21.15.158:0/856078182 wait complete. 2024-02-10T09:31:17.823 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:16.121041+0000 mgr.smithi083.dowdym (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:18.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:17.821116+0000 mon.smithi083 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.158:0/856078182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:19.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:19.561 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:20 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:18.121427+0000 mgr.smithi083.dowdym (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:21.562 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 -- 172.21.15.158:0/1523916301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 msgr2=0x7f33f40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:21.562 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 --2- 172.21.15.158:0/1523916301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f40f8f20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f33e8009fb0 tx=0x7f33e8009b30).stop 2024-02-10T09:31:21.562 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 -- 172.21.15.158:0/1523916301 shutdown_connections 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 --2- 172.21.15.158:0/1523916301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f40f8f20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 -- 172.21.15.158:0/1523916301 >> 172.21.15.158:0/1523916301 conn(0x7f33f40f09a0 msgr2=0x7f33f40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 -- 172.21.15.158:0/1523916301 shutdown_connections 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 -- 172.21.15.158:0/1523916301 wait complete. 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 Processor -- start 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.560+0000 7f33f99ff700 1 -- start start 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f99ff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f99ff700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4106af0 con 0x7f33f40601e0 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f89fd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f89fd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49712/0 (socket says 172.21.15.158:49712) 2024-02-10T09:31:21.563 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f89fd700 1 -- 172.21.15.158:0/3897511747 learned_addr learned my addr 172.21.15.158:0/3897511747 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f89fd700 1 -- 172.21.15.158:0/3897511747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e8005040 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.561+0000 7f33f89fd700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f4106600 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f33e8005010 tx=0x7f33e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.562+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33e8009320 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.562+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f4106cf0 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.562+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33e8013380 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.562+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33e8014e50 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.562+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f4107180 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.562+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33e80263a0 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.563+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f4045ad0 con 0x7f33f40601e0 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.563+0000 7f33f17fa700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f33e402ff10 0x7f33e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:21.565 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.563+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33e8041150 con 0x7f33f40601e0 2024-02-10T09:31:21.566 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.563+0000 7f33f3fff700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f33e402ff10 0x7f33e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:21.566 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.563+0000 7f33f3fff700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f33e402ff10 0x7f33e40323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f33e00096a0 tx=0x7f33e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:21.569 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.568+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33e80126f0 con 0x7f33f40601e0 2024-02-10T09:31:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.774+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33f4108030 con 0x7f33f40601e0 2024-02-10T09:31:21.776 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.775+0000 7f33f17fa700 1 -- 172.21.15.158:0/3897511747 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f33e8012330 con 0x7f33f40601e0 2024-02-10T09:31:21.776 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:21.777 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f33e402ff10 msgr2=0x7f33e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f33e402ff10 0x7f33e40323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f33e00096a0 tx=0x7f33e0006b40).stop 2024-02-10T09:31:21.778 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 msgr2=0x7f33f4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f4106600 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f33e8005010 tx=0x7f33e800b040).stop 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 shutdown_connections 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f33e402ff10 0x7f33e40323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 --2- 172.21.15.158:0/3897511747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f33f40601e0 0x7f33f4106600 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 >> 172.21.15.158:0/3897511747 conn(0x7f33f40f09a0 msgr2=0x7f33f40f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 shutdown_connections 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:21.777+0000 7f33f99ff700 1 -- 172.21.15.158:0/3897511747 wait complete. 2024-02-10T09:31:21.779 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:22.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:20.121789+0000 mgr.smithi083.dowdym (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:22.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:21.776425+0000 mon.smithi083 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.158:0/3897511747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:23.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:23.446 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:22.122161+0000 mgr.smithi083.dowdym (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:25.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 -- 172.21.15.158:0/2954716793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 msgr2=0x7f2cd40f8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:25.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 --2- 172.21.15.158:0/2954716793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd40f8ea0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f2cc4009fb0 tx=0x7f2cc4009b30).stop 2024-02-10T09:31:25.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 -- 172.21.15.158:0/2954716793 shutdown_connections 2024-02-10T09:31:25.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 --2- 172.21.15.158:0/2954716793 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd40f8ea0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 -- 172.21.15.158:0/2954716793 >> 172.21.15.158:0/2954716793 conn(0x7f2cd40f0920 msgr2=0x7f2cd40f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 -- 172.21.15.158:0/2954716793 shutdown_connections 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.234+0000 7f2cd94eb700 1 -- 172.21.15.158:0/2954716793 wait complete. 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd94eb700 1 Processor -- start 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd94eb700 1 -- start start 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd94eb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd41065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd94eb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd4106ad0 con 0x7f2cd4060280 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd41065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd41065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49734/0 (socket says 172.21.15.158:49734) 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd2ffd700 1 -- 172.21.15.158:0/886016530 learned_addr learned my addr 172.21.15.158:0/886016530 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd2ffd700 1 -- 172.21.15.158:0/886016530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4005040 con 0x7f2cd4060280 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.235+0000 7f2cd2ffd700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd41065e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f2cc40089f0 tx=0x7f2cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.236+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cc4009320 con 0x7f2cd4060280 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.236+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cc4013380 con 0x7f2cd4060280 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.236+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd4106cd0 con 0x7f2cd4060280 2024-02-10T09:31:25.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.236+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cc4014e20 con 0x7f2cd4060280 2024-02-10T09:31:25.239 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.236+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd4107160 con 0x7f2cd4060280 2024-02-10T09:31:25.239 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.236+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2cc40095f0 con 0x7f2cd4060280 2024-02-10T09:31:25.239 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.237+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd4045ad0 con 0x7f2cd4060280 2024-02-10T09:31:25.239 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.237+0000 7f2cbb7fe700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2cbc02ff10 0x7f2cbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:25.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.237+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2cc4040f50 con 0x7f2cd4060280 2024-02-10T09:31:25.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.237+0000 7f2cd27fc700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2cbc02ff10 0x7f2cbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:25.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.237+0000 7f2cd27fc700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2cbc02ff10 0x7f2cbc0323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f2cc80096a0 tx=0x7f2cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:25.243 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.241+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cc4014420 con 0x7f2cd4060280 2024-02-10T09:31:25.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.446+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cd4107f60 con 0x7f2cd4060280 2024-02-10T09:31:25.448 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.447+0000 7f2cbb7fe700 1 -- 172.21.15.158:0/886016530 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2cc401d980 con 0x7f2cd4060280 2024-02-10T09:31:25.449 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:25.449 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2cbc02ff10 msgr2=0x7f2cbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2cbc02ff10 0x7f2cbc0323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f2cc80096a0 tx=0x7f2cc8006b40).stop 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 msgr2=0x7f2cd41065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd41065e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f2cc40089f0 tx=0x7f2cc400b040).stop 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 shutdown_connections 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f2cbc02ff10 0x7f2cbc0323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 --2- 172.21.15.158:0/886016530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cd4060280 0x7f2cd41065e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:25.450 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.448+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 >> 172.21.15.158:0/886016530 conn(0x7f2cd40f0920 msgr2=0x7f2cd40f13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:25.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.449+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 shutdown_connections 2024-02-10T09:31:25.451 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:25.449+0000 7f2cd94eb700 1 -- 172.21.15.158:0/886016530 wait complete. 2024-02-10T09:31:25.451 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:26.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:24.122519+0000 mgr.smithi083.dowdym (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:26.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:25.448406+0000 mon.smithi083 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.158:0/886016530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:27.207 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:27.208 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:28.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:28 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:26.122922+0000 mgr.smithi083.dowdym (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 -- 172.21.15.158:0/4226168689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 msgr2=0x7f29340f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 --2- 172.21.15.158:0/4226168689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f4c80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f2928009fb0 tx=0x7f2928009b30).stop 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 -- 172.21.15.158:0/4226168689 shutdown_connections 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 --2- 172.21.15.158:0/4226168689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f4c80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 -- 172.21.15.158:0/4226168689 >> 172.21.15.158:0/4226168689 conn(0x7f29340f02e0 msgr2=0x7f29340f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 -- 172.21.15.158:0/4226168689 shutdown_connections 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 -- 172.21.15.158:0/4226168689 wait complete. 2024-02-10T09:31:29.091 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.088+0000 7f2939f78700 1 Processor -- start 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f2939f78700 1 -- start start 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f2939f78700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f2939f78700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29340f58d0 con 0x7f29340f4860 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f29337fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f29337fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49742/0 (socket says 172.21.15.158:49742) 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f29337fe700 1 -- 172.21.15.158:0/3416104537 learned_addr learned my addr 172.21.15.158:0/3416104537 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f29337fe700 1 -- 172.21.15.158:0/3416104537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2928005040 con 0x7f29340f4860 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.089+0000 7f29337fe700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f7450 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f29280089f0 tx=0x7f292800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.090+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2928009320 con 0x7f29340f4860 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.090+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29340f5a70 con 0x7f29340f4860 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.090+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2928013380 con 0x7f29340f4860 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.090+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2928014e20 con 0x7f29340f4860 2024-02-10T09:31:29.092 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.090+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29340f5f00 con 0x7f29340f4860 2024-02-10T09:31:29.093 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.091+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2928014420 con 0x7f29340f4860 2024-02-10T09:31:29.093 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.091+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2934059100 con 0x7f29340f4860 2024-02-10T09:31:29.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.091+0000 7f291bfff700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f291c02ff10 0x7f291c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:29.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.091+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2928040f00 con 0x7f29340f4860 2024-02-10T09:31:29.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.091+0000 7f2932ffd700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f291c02ff10 0x7f291c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:29.094 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.091+0000 7f2932ffd700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f291c02ff10 0x7f291c0323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f29240096a0 tx=0x7f2924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:29.097 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.095+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29280095f0 con 0x7f29340f4860 2024-02-10T09:31:29.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.301+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2934045ad0 con 0x7f29340f4860 2024-02-10T09:31:29.303 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.302+0000 7f291bfff700 1 -- 172.21.15.158:0/3416104537 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f292801db70 con 0x7f29340f4860 2024-02-10T09:31:29.304 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:29.304 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f291c02ff10 msgr2=0x7f291c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:29.305 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f291c02ff10 0x7f291c0323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f29240096a0 tx=0x7f2924006b40).stop 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 msgr2=0x7f29340f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f7450 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f29280089f0 tx=0x7f292800b040).stop 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 shutdown_connections 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f291c02ff10 0x7f291c0323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 --2- 172.21.15.158:0/3416104537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f29340f4860 0x7f29340f7450 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 >> 172.21.15.158:0/3416104537 conn(0x7f29340f02e0 msgr2=0x7f29340f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 shutdown_connections 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:29.304+0000 7f2939f78700 1 -- 172.21.15.158:0/3416104537 wait complete. 2024-02-10T09:31:29.306 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:30.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:30 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:28.123237+0000 mgr.smithi083.dowdym (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:30.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:30 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:29.303507+0000 mon.smithi083 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.158:0/3416104537' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:31.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:31.015 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:32.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:32 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:30.123578+0000 mgr.smithi083.dowdym (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 -- 172.21.15.158:0/1418476682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 msgr2=0x7f62540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 --2- 172.21.15.158:0/1418476682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f62540f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f6244009fb0 tx=0x7f6244009b30).stop 2024-02-10T09:31:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 -- 172.21.15.158:0/1418476682 shutdown_connections 2024-02-10T09:31:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 --2- 172.21.15.158:0/1418476682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f62540f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 -- 172.21.15.158:0/1418476682 >> 172.21.15.158:0/1418476682 conn(0x7f62540f0b70 msgr2=0x7f62540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:33.065 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 -- 172.21.15.158:0/1418476682 shutdown_connections 2024-02-10T09:31:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 -- 172.21.15.158:0/1418476682 wait complete. 2024-02-10T09:31:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 Processor -- start 2024-02-10T09:31:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.063+0000 7f625c615700 1 -- start start 2024-02-10T09:31:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625c615700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f6254105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:33.066 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625c615700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6254105c00 con 0x7f62540f5410 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625a3b1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f6254105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625a3b1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f6254105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49464/0 (socket says 172.21.15.158:49464) 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625a3b1700 1 -- 172.21.15.158:0/3854990327 learned_addr learned my addr 172.21.15.158:0/3854990327 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625a3b1700 1 -- 172.21.15.158:0/3854990327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6244005040 con 0x7f62540f5410 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f625a3b1700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f6254105710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6244005010 tx=0x7f624400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.064+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6244009320 con 0x7f62540f5410 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.065+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6254105e00 con 0x7f62540f5410 2024-02-10T09:31:33.068 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.065+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6244013380 con 0x7f62540f5410 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.065+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6244014e50 con 0x7f62540f5410 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.065+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6254106290 con 0x7f62540f5410 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.065+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62440095f0 con 0x7f62540f5410 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.065+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6254045ad0 con 0x7f62540f5410 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.066+0000 7f624affd700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f624002ff10 0x7f62400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.066+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6244042c70 con 0x7f62540f5410 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.066+0000 7f6259bb0700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f624002ff10 0x7f62400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:33.069 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.066+0000 7f6259bb0700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f624002ff10 0x7f62400323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f62500096a0 tx=0x7f6250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:33.071 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.070+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6244014420 con 0x7f62540f5410 2024-02-10T09:31:33.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.276+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6254107140 con 0x7f62540f5410 2024-02-10T09:31:33.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.277+0000 7f624affd700 1 -- 172.21.15.158:0/3854990327 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6244011290 con 0x7f62540f5410 2024-02-10T09:31:33.278 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:33.279 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:33.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f624002ff10 msgr2=0x7f62400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:33.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f624002ff10 0x7f62400323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f62500096a0 tx=0x7f6250006b40).stop 2024-02-10T09:31:33.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 msgr2=0x7f6254105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:33.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f6254105710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f6244005010 tx=0x7f624400b040).stop 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 shutdown_connections 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f624002ff10 0x7f62400323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 --2- 172.21.15.158:0/3854990327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62540f5410 0x7f6254105710 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 >> 172.21.15.158:0/3854990327 conn(0x7f62540f0b70 msgr2=0x7f62540f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 shutdown_connections 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:33.279+0000 7f625c615700 1 -- 172.21.15.158:0/3854990327 wait complete. 2024-02-10T09:31:33.281 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:34.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:34 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:32.123900+0000 mgr.smithi083.dowdym (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:34.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:34 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:33.278401+0000 mon.smithi083 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.158:0/3854990327' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:35.058 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:35.058 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:36.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:34.124272+0000 mgr.smithi083.dowdym (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:37.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 -- 172.21.15.158:0/3894148038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 msgr2=0x7f9f000f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:37.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 --2- 172.21.15.158:0/3894148038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f000f9170 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f9ef0009fb0 tx=0x7f9ef0009b30).stop 2024-02-10T09:31:37.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 -- 172.21.15.158:0/3894148038 shutdown_connections 2024-02-10T09:31:37.081 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 --2- 172.21.15.158:0/3894148038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f000f9170 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 -- 172.21.15.158:0/3894148038 >> 172.21.15.158:0/3894148038 conn(0x7f9f000f0b70 msgr2=0x7f9f000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 -- 172.21.15.158:0/3894148038 shutdown_connections 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.078+0000 7f9f05e06700 1 -- 172.21.15.158:0/3894148038 wait complete. 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9f05e06700 1 Processor -- start 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9f05e06700 1 -- start start 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9f05e06700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f001067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9f05e06700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f00106c90 con 0x7f9f00060200 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9eff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f001067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9eff7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f001067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49472/0 (socket says 172.21.15.158:49472) 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.079+0000 7f9eff7fe700 1 -- 172.21.15.158:0/1323897804 learned_addr learned my addr 172.21.15.158:0/1323897804 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9eff7fe700 1 -- 172.21.15.158:0/1323897804 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef0005040 con 0x7f9f00060200 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9eff7fe700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f001067a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9ef0009830 tx=0x7f9ef000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:37.082 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef0009320 con 0x7f9f00060200 2024-02-10T09:31:37.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f00106e90 con 0x7f9f00060200 2024-02-10T09:31:37.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ef0013380 con 0x7f9f00060200 2024-02-10T09:31:37.083 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef0014420 con 0x7f9f00060200 2024-02-10T09:31:37.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.080+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f00107320 con 0x7f9f00060200 2024-02-10T09:31:37.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.081+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ef001e420 con 0x7f9f00060200 2024-02-10T09:31:37.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.081+0000 7f9efcff9700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9eec02ff60 0x7f9eec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:37.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.081+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ef0040f40 con 0x7f9f00060200 2024-02-10T09:31:37.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.081+0000 7f9ef6ffd700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9eec02ff60 0x7f9eec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:37.084 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.081+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f00059100 con 0x7f9f00060200 2024-02-10T09:31:37.085 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.082+0000 7f9ef6ffd700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9eec02ff60 0x7f9eec032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9ee80096a0 tx=0x7f9ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:37.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.086+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ef00095f0 con 0x7f9f00060200 2024-02-10T09:31:37.293 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.292+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f00045ad0 con 0x7f9f00060200 2024-02-10T09:31:37.294 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.293+0000 7f9efcff9700 1 -- 172.21.15.158:0/1323897804 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ef000fe60 con 0x7f9f00060200 2024-02-10T09:31:37.294 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:37.294 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:37.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9eec02ff60 msgr2=0x7f9eec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:37.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9eec02ff60 0x7f9eec032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f9ee80096a0 tx=0x7f9ee8006b40).stop 2024-02-10T09:31:37.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 msgr2=0x7f9f001067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:37.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f001067a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9ef0009830 tx=0x7f9ef000b040).stop 2024-02-10T09:31:37.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 shutdown_connections 2024-02-10T09:31:37.296 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9eec02ff60 0x7f9eec032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:37.297 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 --2- 172.21.15.158:0/1323897804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00060200 0x7f9f001067a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:37.297 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 >> 172.21.15.158:0/1323897804 conn(0x7f9f000f0b70 msgr2=0x7f9f000f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:37.297 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 shutdown_connections 2024-02-10T09:31:37.297 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:37.294+0000 7f9f05e06700 1 -- 172.21.15.158:0/1323897804 wait complete. 2024-02-10T09:31:37.297 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:38 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:36.124659+0000 mgr.smithi083.dowdym (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:38 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:37.294045+0000 mon.smithi083 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.158:0/1323897804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:39.015 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:39.015 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:40.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:40 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:38.124972+0000 mgr.smithi083.dowdym (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 -- 172.21.15.158:0/775532054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 msgr2=0x7f9b940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 --2- 172.21.15.158:0/775532054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b940f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f9b88009fb0 tx=0x7f9b88009b30).stop 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 -- 172.21.15.158:0/775532054 shutdown_connections 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 --2- 172.21.15.158:0/775532054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b940f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 -- 172.21.15.158:0/775532054 >> 172.21.15.158:0/775532054 conn(0x7f9b940f0b70 msgr2=0x7f9b940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 -- 172.21.15.158:0/775532054 shutdown_connections 2024-02-10T09:31:41.008 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.006+0000 7f9b994fe700 1 -- 172.21.15.158:0/775532054 wait complete. 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b994fe700 1 Processor -- start 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b994fe700 1 -- start start 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b994fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b941056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b994fe700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b94105bd0 con 0x7f9b940f5410 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b92ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b941056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b92ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b941056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49178/0 (socket says 172.21.15.158:49178) 2024-02-10T09:31:41.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.007+0000 7f9b92ffd700 1 -- 172.21.15.158:0/4250283941 learned_addr learned my addr 172.21.15.158:0/4250283941 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b92ffd700 1 -- 172.21.15.158:0/4250283941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b88005040 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b92ffd700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b941056e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f9b88005010 tx=0x7f9b8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b88009320 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b94105dd0 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b88013380 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b88014e50 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.008+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b94106260 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.009+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b88014420 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.009+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b94045ad0 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.009+0000 7f9b7b7fe700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b7c02ff10 0x7f9b7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.009+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b88040ff0 con 0x7f9b940f5410 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.009+0000 7f9b927fc700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b7c02ff10 0x7f9b7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:41.012 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.010+0000 7f9b927fc700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b7c02ff10 0x7f9b7c0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f9b840096a0 tx=0x7f9b84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:41.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.014+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b8806d2b0 con 0x7f9b940f5410 2024-02-10T09:31:41.220 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.219+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b94107110 con 0x7f9b940f5410 2024-02-10T09:31:41.220 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.219+0000 7f9b7b7fe700 1 -- 172.21.15.158:0/4250283941 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b88014790 con 0x7f9b940f5410 2024-02-10T09:31:41.221 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:41.221 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:41.222 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b7c02ff10 msgr2=0x7f9b7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b7c02ff10 0x7f9b7c0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f9b840096a0 tx=0x7f9b84006b40).stop 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 msgr2=0x7f9b941056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b941056e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f9b88005010 tx=0x7f9b8800b040).stop 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 shutdown_connections 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f9b7c02ff10 0x7f9b7c0323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 --2- 172.21.15.158:0/4250283941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b940f5410 0x7f9b941056e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 >> 172.21.15.158:0/4250283941 conn(0x7f9b940f0b70 msgr2=0x7f9b940f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 shutdown_connections 2024-02-10T09:31:41.223 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:41.221+0000 7f9b994fe700 1 -- 172.21.15.158:0/4250283941 wait complete. 2024-02-10T09:31:41.224 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:42 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:40.125228+0000 mgr.smithi083.dowdym (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:42.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:42 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:41.220840+0000 mon.smithi083 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.158:0/4250283941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:42.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:42.861 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:44.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:42.125611+0000 mgr.smithi083.dowdym (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:44.783 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 -- 172.21.15.158:0/1045581233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 msgr2=0x7f0dd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 --2- 172.21.15.158:0/1045581233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd80f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f0dcc009fb0 tx=0x7f0dcc009b30).stop 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 -- 172.21.15.158:0/1045581233 shutdown_connections 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 --2- 172.21.15.158:0/1045581233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd80f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 -- 172.21.15.158:0/1045581233 >> 172.21.15.158:0/1045581233 conn(0x7f0dd80f1c70 msgr2=0x7f0dd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 -- 172.21.15.158:0/1045581233 shutdown_connections 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.780+0000 7f0ddcafd700 1 -- 172.21.15.158:0/1045581233 wait complete. 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0ddcafd700 1 Processor -- start 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0ddcafd700 1 -- start start 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0ddcafd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:44.784 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0ddcafd700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd8069e20 con 0x7f0dd80f6510 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0dd659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0dd659c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49204/0 (socket says 172.21.15.158:49204) 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.781+0000 7f0dd659c700 1 -- 172.21.15.158:0/376301725 learned_addr learned my addr 172.21.15.158:0/376301725 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0dd659c700 1 -- 172.21.15.158:0/376301725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc005040 con 0x7f0dd80f6510 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0dd659c700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd806b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0dcc005010 tx=0x7f0dcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dcc009320 con 0x7f0dd80f6510 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd8069fc0 con 0x7f0dd80f6510 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dcc013380 con 0x7f0dd80f6510 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dcc014e50 con 0x7f0dd80f6510 2024-02-10T09:31:44.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.782+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd806a450 con 0x7f0dd80f6510 2024-02-10T09:31:44.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.783+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0dcc0095f0 con 0x7f0dd80f6510 2024-02-10T09:31:44.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.783+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd8059100 con 0x7f0dd80f6510 2024-02-10T09:31:44.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.783+0000 7f0dc6ffd700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dc002ff10 0x7f0dc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:44.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.783+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0dcc040e70 con 0x7f0dd80f6510 2024-02-10T09:31:44.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.783+0000 7f0dd5d9b700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dc002ff10 0x7f0dc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:44.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.783+0000 7f0dd5d9b700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dc002ff10 0x7f0dc00323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0dc80096a0 tx=0x7f0dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:44.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.787+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0dcc014420 con 0x7f0dd80f6510 2024-02-10T09:31:44.990 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.989+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0dd80fc0a0 con 0x7f0dd80f6510 2024-02-10T09:31:44.991 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.990+0000 7f0dc6ffd700 1 -- 172.21.15.158:0/376301725 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0dcc010230 con 0x7f0dd80f6510 2024-02-10T09:31:44.991 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:44.991 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dc002ff10 msgr2=0x7f0dc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dc002ff10 0x7f0dc00323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0dc80096a0 tx=0x7f0dc8006b40).stop 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 msgr2=0x7f0dd806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd806b9a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0dcc005010 tx=0x7f0dcc00b040).stop 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 shutdown_connections 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f0dc002ff10 0x7f0dc00323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 --2- 172.21.15.158:0/376301725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0dd80f6510 0x7f0dd806b9a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 >> 172.21.15.158:0/376301725 conn(0x7f0dd80f1c70 msgr2=0x7f0dd80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 shutdown_connections 2024-02-10T09:31:44.993 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:44.991+0000 7f0ddcafd700 1 -- 172.21.15.158:0/376301725 wait complete. 2024-02-10T09:31:44.994 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:46.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:44.125903+0000 mgr.smithi083.dowdym (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:46.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:44.991261+0000 mon.smithi083 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.158:0/376301725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:46.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:46.682 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:48.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:46.126172+0000 mgr.smithi083.dowdym (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:48.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 -- 172.21.15.158:0/3489610245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 msgr2=0x7f79780f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:48.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 --2- 172.21.15.158:0/3489610245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f79780f8f20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f7968009fb0 tx=0x7f7968009b30).stop 2024-02-10T09:31:48.596 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 -- 172.21.15.158:0/3489610245 shutdown_connections 2024-02-10T09:31:48.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 --2- 172.21.15.158:0/3489610245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f79780f8f20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:48.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 -- 172.21.15.158:0/3489610245 >> 172.21.15.158:0/3489610245 conn(0x7f79780f09a0 msgr2=0x7f79780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:48.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 -- 172.21.15.158:0/3489610245 shutdown_connections 2024-02-10T09:31:48.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 -- 172.21.15.158:0/3489610245 wait complete. 2024-02-10T09:31:48.597 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 Processor -- start 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.595+0000 7f797cd6e700 1 -- start start 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f797cd6e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f7978106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f797cd6e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7978106af0 con 0x7f79780601e0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f79777fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f7978106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f79777fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f7978106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:49210/0 (socket says 172.21.15.158:49210) 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f79777fe700 1 -- 172.21.15.158:0/3142780609 learned_addr learned my addr 172.21.15.158:0/3142780609 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f79777fe700 1 -- 172.21.15.158:0/3142780609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7968005040 con 0x7f79780601e0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f79777fe700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f7978106600 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7968005010 tx=0x7f796800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7968009320 con 0x7f79780601e0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7968013380 con 0x7f79780601e0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7978106cf0 con 0x7f79780601e0 2024-02-10T09:31:48.599 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7968014e50 con 0x7f79780601e0 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.596+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7978107180 con 0x7f79780601e0 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.597+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79680263a0 con 0x7f79780601e0 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.597+0000 7f795ffff700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f796002ff10 0x7f79600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.597+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7968040f40 con 0x7f79780601e0 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.597+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7978045ad0 con 0x7f79780601e0 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.598+0000 7f7976ffd700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f796002ff10 0x7f79600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:48.600 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.598+0000 7f7976ffd700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f796002ff10 0x7f79600323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f796c0096a0 tx=0x7f796c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:48.603 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.602+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7968012920 con 0x7f79780601e0 2024-02-10T09:31:48.809 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.808+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7978108140 con 0x7f79780601e0 2024-02-10T09:31:48.810 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.809+0000 7f795ffff700 1 -- 172.21.15.158:0/3142780609 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f79680126f0 con 0x7f79780601e0 2024-02-10T09:31:48.810 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:48.810 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f796002ff10 msgr2=0x7f79600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f796002ff10 0x7f79600323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f796c0096a0 tx=0x7f796c006b40).stop 2024-02-10T09:31:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 msgr2=0x7f7978106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f7978106600 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7968005010 tx=0x7f796800b040).stop 2024-02-10T09:31:48.812 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 shutdown_connections 2024-02-10T09:31:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f796002ff10 0x7f79600323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 --2- 172.21.15.158:0/3142780609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79780601e0 0x7f7978106600 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 >> 172.21.15.158:0/3142780609 conn(0x7f79780f09a0 msgr2=0x7f79780f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 shutdown_connections 2024-02-10T09:31:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:48.811+0000 7f797cd6e700 1 -- 172.21.15.158:0/3142780609 wait complete. 2024-02-10T09:31:48.813 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:50.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:48.126494+0000 mgr.smithi083.dowdym (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:50.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:48.810341+0000 mon.smithi083 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.158:0/3142780609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:50.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:50.545 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:52.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:50.126858+0000 mgr.smithi083.dowdym (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 -- 172.21.15.158:0/208425244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 msgr2=0x7fe2480f4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 --2- 172.21.15.158:0/208425244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe2480f4b10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fe244009fb0 tx=0x7fe244009b30).stop 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 -- 172.21.15.158:0/208425244 shutdown_connections 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 --2- 172.21.15.158:0/208425244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe2480f4b10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 -- 172.21.15.158:0/208425244 >> 172.21.15.158:0/208425244 conn(0x7fe2480f0210 msgr2=0x7fe2480f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 -- 172.21.15.158:0/208425244 shutdown_connections 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.548+0000 7fe24f2f0700 1 -- 172.21.15.158:0/208425244 wait complete. 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24f2f0700 1 Processor -- start 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24f2f0700 1 -- start start 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24f2f0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe248106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:52.551 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24f2f0700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe248106c50 con 0x7fe2480f46f0 2024-02-10T09:31:52.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24d08c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe248106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:52.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24d08c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe248106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:37586/0 (socket says 172.21.15.158:37586) 2024-02-10T09:31:52.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24d08c700 1 -- 172.21.15.158:0/1768086063 learned_addr learned my addr 172.21.15.158:0/1768086063 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:52.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.549+0000 7fe24d08c700 1 -- 172.21.15.158:0/1768086063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244005040 con 0x7fe2480f46f0 2024-02-10T09:31:52.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.550+0000 7fe24d08c700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe248106760 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fe244005010 tx=0x7fe24400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:52.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.550+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe244009320 con 0x7fe2480f46f0 2024-02-10T09:31:52.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.550+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe248106e50 con 0x7fe2480f46f0 2024-02-10T09:31:52.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.550+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe244013380 con 0x7fe2480f46f0 2024-02-10T09:31:52.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.550+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe244014e50 con 0x7fe2480f46f0 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.550+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2481072e0 con 0x7fe2480f46f0 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.551+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe244014420 con 0x7fe2480f46f0 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.551+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe248045ad0 con 0x7fe2480f46f0 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.551+0000 7fe23dffb700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe23402ff10 0x7fe2340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.551+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe244040f70 con 0x7fe2480f46f0 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.551+0000 7fe24c88b700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe23402ff10 0x7fe2340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:52.554 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.552+0000 7fe24c88b700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe23402ff10 0x7fe2340323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe2380096a0 tx=0x7fe238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:52.557 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.556+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2440095f0 con 0x7fe2480f46f0 2024-02-10T09:31:52.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.762+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2480fc030 con 0x7fe2480f46f0 2024-02-10T09:31:52.765 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.763+0000 7fe23dffb700 1 -- 172.21.15.158:0/1768086063 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe24401db70 con 0x7fe2480f46f0 2024-02-10T09:31:52.765 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:52.765 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:52.766 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe23402ff10 msgr2=0x7fe2340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:52.766 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe23402ff10 0x7fe2340323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fe2380096a0 tx=0x7fe238006b40).stop 2024-02-10T09:31:52.766 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 msgr2=0x7fe248106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe248106760 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fe244005010 tx=0x7fe24400b040).stop 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 shutdown_connections 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe23402ff10 0x7fe2340323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 --2- 172.21.15.158:0/1768086063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2480f46f0 0x7fe248106760 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 >> 172.21.15.158:0/1768086063 conn(0x7fe2480f0210 msgr2=0x7fe2480f0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 shutdown_connections 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:52.765+0000 7fe24f2f0700 1 -- 172.21.15.158:0/1768086063 wait complete. 2024-02-10T09:31:52.767 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:54.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:52.127107+0000 mgr.smithi083.dowdym (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:54.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:52.764656+0000 mon.smithi083 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.158:0/1768086063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:54.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:54.408 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:31:56.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:54.127456+0000 mgr.smithi083.dowdym (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:56.325 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 -- 172.21.15.158:0/1662583232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 msgr2=0x7f59c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:56.325 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 --2- 172.21.15.158:0/1662583232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c00f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f59b4009fb0 tx=0x7f59b4009b30).stop 2024-02-10T09:31:56.325 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 -- 172.21.15.158:0/1662583232 shutdown_connections 2024-02-10T09:31:56.325 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 --2- 172.21.15.158:0/1662583232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c00f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:56.325 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 -- 172.21.15.158:0/1662583232 >> 172.21.15.158:0/1662583232 conn(0x7f59c00f1c70 msgr2=0x7f59c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:56.325 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 -- 172.21.15.158:0/1662583232 shutdown_connections 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.322+0000 7f59c6b69700 1 -- 172.21.15.158:0/1662583232 wait complete. 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.323+0000 7f59c6b69700 1 Processor -- start 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.323+0000 7f59c6b69700 1 -- start start 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.323+0000 7f59c6b69700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.323+0000 7f59c6b69700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c0069e20 con 0x7f59c00f6510 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.323+0000 7f59c4905700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59c4905700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:37608/0 (socket says 172.21.15.158:37608) 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59c4905700 1 -- 172.21.15.158:0/328056583 learned_addr learned my addr 172.21.15.158:0/328056583 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59c4905700 1 -- 172.21.15.158:0/328056583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4005040 con 0x7f59c00f6510 2024-02-10T09:31:56.326 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59c4905700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c006b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f59b4005010 tx=0x7f59b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:56.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b4009320 con 0x7f59c00f6510 2024-02-10T09:31:56.327 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c0069fc0 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59b4013380 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b4014e50 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.324+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c006a450 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.325+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59b40263a0 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.325+0000 7f59bd7fa700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59b002ff10 0x7f59b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.325+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59b4040fa0 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.325+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c0059100 con 0x7f59c00f6510 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.325+0000 7f59bffff700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59b002ff10 0x7f59b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:31:56.328 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.326+0000 7f59bffff700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59b002ff10 0x7f59b00323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f59ac0096a0 tx=0x7f59ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:31:56.331 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.331+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59b4012920 con 0x7f59c00f6510 2024-02-10T09:31:56.538 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.537+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59c00fc0a0 con 0x7f59c00f6510 2024-02-10T09:31:56.539 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.537+0000 7f59bd7fa700 1 -- 172.21.15.158:0/328056583 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59b4012330 con 0x7f59c00f6510 2024-02-10T09:31:56.539 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:31:56.539 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:31:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59b002ff10 msgr2=0x7f59b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59b002ff10 0x7f59b00323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f59ac0096a0 tx=0x7f59ac006b40).stop 2024-02-10T09:31:56.540 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 msgr2=0x7f59c006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c006b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f59b4005010 tx=0x7f59b400b040).stop 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 shutdown_connections 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59b002ff10 0x7f59b00323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 --2- 172.21.15.158:0/328056583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59c00f6510 0x7f59c006b9a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 >> 172.21.15.158:0/328056583 conn(0x7f59c00f1c70 msgr2=0x7f59c00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 shutdown_connections 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:31:56.539+0000 7f59c6b69700 1 -- 172.21.15.158:0/328056583 wait complete. 2024-02-10T09:31:56.541 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:31:57.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:56 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:31:56.538804+0000 mon.smithi083 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.158:0/328056583' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:31:58.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:56.127822+0000 mgr.smithi083.dowdym (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:31:58.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:31:58.215 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.023+0000 7fb4712cb700 1 -- 172.21.15.158:0/794543534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 msgr2=0x7fb46c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.023+0000 7fb4712cb700 1 --2- 172.21.15.158:0/794543534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c0f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb45c009fb0 tx=0x7fb45c009b30).stop 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.023+0000 7fb4712cb700 1 -- 172.21.15.158:0/794543534 shutdown_connections 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.023+0000 7fb4712cb700 1 --2- 172.21.15.158:0/794543534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c0f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.023+0000 7fb4712cb700 1 -- 172.21.15.158:0/794543534 >> 172.21.15.158:0/794543534 conn(0x7fb46c0f0b70 msgr2=0x7fb46c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.024+0000 7fb4712cb700 1 -- 172.21.15.158:0/794543534 shutdown_connections 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.024+0000 7fb4712cb700 1 -- 172.21.15.158:0/794543534 wait complete. 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.024+0000 7fb4712cb700 1 Processor -- start 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.024+0000 7fb4712cb700 1 -- start start 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.024+0000 7fb4712cb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:00.026 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.024+0000 7fb4712cb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c106d00 con 0x7fb46c0f5410 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb46affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb46affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:35362/0 (socket says 172.21.15.158:35362) 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb46affd700 1 -- 172.21.15.158:0/2759966556 learned_addr learned my addr 172.21.15.158:0/2759966556 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb46affd700 1 -- 172.21.15.158:0/2759966556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c005040 con 0x7fb46c0f5410 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb46affd700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c106810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb45c005010 tx=0x7fb45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c009320 con 0x7fb46c0f5410 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c106f00 con 0x7fb46c0f5410 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb45c013380 con 0x7fb46c0f5410 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb45c014e50 con 0x7fb46c0f5410 2024-02-10T09:32:00.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.025+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c107390 con 0x7fb46c0f5410 2024-02-10T09:32:00.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:31:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:31:58.128091+0000 mgr.smithi083.dowdym (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:00.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.026+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb45c014420 con 0x7fb46c0f5410 2024-02-10T09:32:00.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.026+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c045ad0 con 0x7fb46c0f5410 2024-02-10T09:32:00.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.026+0000 7fb4537fe700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb45402ff10 0x7fb4540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:00.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.026+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb45c040e60 con 0x7fb46c0f5410 2024-02-10T09:32:00.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.026+0000 7fb46a7fc700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb45402ff10 0x7fb4540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:00.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.027+0000 7fb46a7fc700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb45402ff10 0x7fb4540323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4600096a0 tx=0x7fb460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:00.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.031+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb45c0095f0 con 0x7fb46c0f5410 2024-02-10T09:32:00.237 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.237+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb46c0fafa0 con 0x7fb46c0f5410 2024-02-10T09:32:00.238 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.238+0000 7fb4537fe700 1 -- 172.21.15.158:0/2759966556 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb45c01db70 con 0x7fb46c0f5410 2024-02-10T09:32:00.238 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:00.238 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb45402ff10 msgr2=0x7fb4540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb45402ff10 0x7fb4540323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4600096a0 tx=0x7fb460006b40).stop 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 msgr2=0x7fb46c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c106810 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb45c005010 tx=0x7fb45c00b040).stop 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 shutdown_connections 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb45402ff10 0x7fb4540323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:00.240 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 --2- 172.21.15.158:0/2759966556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb46c0f5410 0x7fb46c106810 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:00.241 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.239+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 >> 172.21.15.158:0/2759966556 conn(0x7fb46c0f0b70 msgr2=0x7fb46c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:00.241 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.240+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 shutdown_connections 2024-02-10T09:32:00.241 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:00.240+0000 7fb4712cb700 1 -- 172.21.15.158:0/2759966556 wait complete. 2024-02-10T09:32:00.241 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:01.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:00 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:00.238046+0000 mon.smithi083 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.158:0/2759966556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:01.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:01.802 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:02.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:00.128390+0000 mgr.smithi083.dowdym (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 -- 172.21.15.158:0/3304850310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 msgr2=0x7f70b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 --2- 172.21.15.158:0/3304850310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b80f6930 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f70ac009fb0 tx=0x7f70ac009b30).stop 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 -- 172.21.15.158:0/3304850310 shutdown_connections 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 --2- 172.21.15.158:0/3304850310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b80f6930 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 -- 172.21.15.158:0/3304850310 >> 172.21.15.158:0/3304850310 conn(0x7f70b80f1c70 msgr2=0x7f70b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 -- 172.21.15.158:0/3304850310 shutdown_connections 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 -- 172.21.15.158:0/3304850310 wait complete. 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 Processor -- start 2024-02-10T09:32:03.826 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.824+0000 7f70bdc86700 1 -- start start 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70bdc86700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70bdc86700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b8069e20 con 0x7f70b80f6510 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70b77fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70b77fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:35376/0 (socket says 172.21.15.158:35376) 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70b77fe700 1 -- 172.21.15.158:0/2682342084 learned_addr learned my addr 172.21.15.158:0/2682342084 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70b77fe700 1 -- 172.21.15.158:0/2682342084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac005040 con 0x7f70b80f6510 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70b77fe700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b806b9a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f70ac005010 tx=0x7f70ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70ac009320 con 0x7f70b80f6510 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b8069fc0 con 0x7f70b80f6510 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70ac013380 con 0x7f70b80f6510 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.825+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70ac014e50 con 0x7f70b80f6510 2024-02-10T09:32:03.827 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.826+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b806a450 con 0x7f70b80f6510 2024-02-10T09:32:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.826+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70ac014420 con 0x7f70b80f6510 2024-02-10T09:32:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.826+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b8059100 con 0x7f70b80f6510 2024-02-10T09:32:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.826+0000 7f709ffff700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70a002ff10 0x7f70a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:03.828 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.826+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70ac040e30 con 0x7f70b80f6510 2024-02-10T09:32:03.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.827+0000 7f70b6ffd700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70a002ff10 0x7f70a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:03.829 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.827+0000 7f70b6ffd700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70a002ff10 0x7f70a00323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f70a80096a0 tx=0x7f70a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:03.831 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:03.831+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70ac0095f0 con 0x7f70b80f6510 2024-02-10T09:32:04.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:02.128756+0000 mgr.smithi083.dowdym (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:04.043 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.043+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70b80fc0a0 con 0x7f70b80f6510 2024-02-10T09:32:04.043 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.043+0000 7f709ffff700 1 -- 172.21.15.158:0/2682342084 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f70ac01db70 con 0x7f70b80f6510 2024-02-10T09:32:04.044 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:04.044 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:04.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70a002ff10 msgr2=0x7f70a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:04.045 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70a002ff10 0x7f70a00323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f70a80096a0 tx=0x7f70a8006b40).stop 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 msgr2=0x7f70b806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b806b9a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f70ac005010 tx=0x7f70ac00b040).stop 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 shutdown_connections 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f70a002ff10 0x7f70a00323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 --2- 172.21.15.158:0/2682342084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f70b80f6510 0x7f70b806b9a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 >> 172.21.15.158:0/2682342084 conn(0x7f70b80f1c70 msgr2=0x7f70b80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 shutdown_connections 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:04.045+0000 7f70bdc86700 1 -- 172.21.15.158:0/2682342084 wait complete. 2024-02-10T09:32:04.046 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:05.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:04.043711+0000 mon.smithi083 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.158:0/2682342084' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:05.685 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:05.686 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:06.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:04.129070+0000 mgr.smithi083.dowdym (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:07.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 -- 172.21.15.158:0/1821948899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 msgr2=0x7f168c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:07.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 --2- 172.21.15.158:0/1821948899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f9190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f167c009fb0 tx=0x7f167c009b30).stop 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 -- 172.21.15.158:0/1821948899 shutdown_connections 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 --2- 172.21.15.158:0/1821948899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f9190 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 -- 172.21.15.158:0/1821948899 >> 172.21.15.158:0/1821948899 conn(0x7f168c0f0b70 msgr2=0x7f168c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 -- 172.21.15.158:0/1821948899 shutdown_connections 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.794+0000 7f1692341700 1 -- 172.21.15.158:0/1821948899 wait complete. 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f1692341700 1 Processor -- start 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f1692341700 1 -- start start 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f1692341700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:07.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f1692341700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0f5090 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f168bfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f168bfff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:35388/0 (socket says 172.21.15.158:35388) 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.795+0000 7f168bfff700 1 -- 172.21.15.158:0/3259238556 learned_addr learned my addr 172.21.15.158:0/3259238556 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f168bfff700 1 -- 172.21.15.158:0/3259238556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c005040 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f168bfff700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f6c10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f167c007a10 tx=0x7f167c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167c018070 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c0f5230 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f167c013410 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167c01c3c0 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.796+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c0f56c0 con 0x7f168c0601e0 2024-02-10T09:32:07.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.797+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f167c013ca0 con 0x7f168c0601e0 2024-02-10T09:32:07.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.797+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c059100 con 0x7f168c0601e0 2024-02-10T09:32:07.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.797+0000 7f1688ff9700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f167402ffb0 0x7f1674032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:07.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.797+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f167c041540 con 0x7f168c0601e0 2024-02-10T09:32:07.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.797+0000 7f168b7fe700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f167402ffb0 0x7f1674032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:07.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.798+0000 7f168b7fe700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f167402ffb0 0x7f1674032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f16800096a0 tx=0x7f1680006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:07.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:07.802+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f167c01d420 con 0x7f168c0601e0 2024-02-10T09:32:08.010 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.010+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f168c0fc0a0 con 0x7f168c0601e0 2024-02-10T09:32:08.011 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.010+0000 7f1688ff9700 1 -- 172.21.15.158:0/3259238556 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f167c025ce0 con 0x7f168c0601e0 2024-02-10T09:32:08.011 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:08.011 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f167402ffb0 msgr2=0x7f1674032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f167402ffb0 0x7f1674032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f16800096a0 tx=0x7f1680006b40).stop 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 msgr2=0x7f168c0f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f6c10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f167c007a10 tx=0x7f167c009b30).stop 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 shutdown_connections 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f167402ffb0 0x7f1674032470 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 --2- 172.21.15.158:0/3259238556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f168c0601e0 0x7f168c0f6c10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 >> 172.21.15.158:0/3259238556 conn(0x7f168c0f0b70 msgr2=0x7f168c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 shutdown_connections 2024-02-10T09:32:08.013 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:08.012+0000 7f1692341700 1 -- 172.21.15.158:0/3259238556 wait complete. 2024-02-10T09:32:08.014 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:08.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:06.129430+0000 mgr.smithi083.dowdym (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:09.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:08.010834+0000 mon.smithi083 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.158:0/3259238556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:09.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:09.719 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:10.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:08.129703+0000 mgr.smithi083.dowdym (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:11.586 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 -- 172.21.15.158:0/4205513663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 msgr2=0x7fef4c0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:11.586 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 --2- 172.21.15.158:0/4205513663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c0f9170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fef3c009fb0 tx=0x7fef3c009b30).stop 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 -- 172.21.15.158:0/4205513663 shutdown_connections 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 --2- 172.21.15.158:0/4205513663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c0f9170 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 -- 172.21.15.158:0/4205513663 >> 172.21.15.158:0/4205513663 conn(0x7fef4c0f0b70 msgr2=0x7fef4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 -- 172.21.15.158:0/4205513663 shutdown_connections 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.585+0000 7fef52084700 1 -- 172.21.15.158:0/4205513663 wait complete. 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.586+0000 7fef52084700 1 Processor -- start 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.586+0000 7fef52084700 1 -- start start 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.586+0000 7fef52084700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c108990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:11.587 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.586+0000 7fef52084700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4c108e80 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef4b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef4b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:55544/0 (socket says 172.21.15.158:55544) 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef4b7fe700 1 -- 172.21.15.158:0/3019945610 learned_addr learned my addr 172.21.15.158:0/3019945610 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef4b7fe700 1 -- 172.21.15.158:0/3019945610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c005040 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef4b7fe700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c108990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fef3c0089f0 tx=0x7fef3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c009320 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4c109080 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef3c013380 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef3c014e20 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.587+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4c109510 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.588+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef3c0095f0 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.588+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4c045ad0 con 0x7fef4c060200 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.588+0000 7fef43fff700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fef3802ff10 0x7fef380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:11.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.588+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef3c040ec0 con 0x7fef4c060200 2024-02-10T09:32:11.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.588+0000 7fef4affd700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fef3802ff10 0x7fef380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:11.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.589+0000 7fef4affd700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fef3802ff10 0x7fef380323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fef340096a0 tx=0x7fef34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:11.593 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.593+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef3c014420 con 0x7fef4c060200 2024-02-10T09:32:11.800 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.799+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef4c10a360 con 0x7fef4c060200 2024-02-10T09:32:11.800 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.800+0000 7fef43fff700 1 -- 172.21.15.158:0/3019945610 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fef3c010230 con 0x7fef4c060200 2024-02-10T09:32:11.801 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:11.801 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:11.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fef3802ff10 msgr2=0x7fef380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:11.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fef3802ff10 0x7fef380323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fef340096a0 tx=0x7fef34006b40).stop 2024-02-10T09:32:11.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 msgr2=0x7fef4c108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:11.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c108990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fef3c0089f0 tx=0x7fef3c00b040).stop 2024-02-10T09:32:11.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 shutdown_connections 2024-02-10T09:32:11.802 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fef3802ff10 0x7fef380323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:11.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 --2- 172.21.15.158:0/3019945610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef4c060200 0x7fef4c108990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:11.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 >> 172.21.15.158:0/3019945610 conn(0x7fef4c0f0b70 msgr2=0x7fef4c0f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:11.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 shutdown_connections 2024-02-10T09:32:11.803 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:11.802+0000 7fef52084700 1 -- 172.21.15.158:0/3019945610 wait complete. 2024-02-10T09:32:11.803 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:12.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:10.129954+0000 mgr.smithi083.dowdym (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:13.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:12 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:11.800679+0000 mon.smithi083 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.158:0/3019945610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:13.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:13.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:14.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:12.130288+0000 mgr.smithi083.dowdym (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:14.176373+0000 mon.smithi083 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:32:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:14.183908+0000 mon.smithi083 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:32:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 -- 172.21.15.158:0/2572389342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 msgr2=0x7f69700f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 --2- 172.21.15.158:0/2572389342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69700f9150 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f6960009fb0 tx=0x7f6960009b30).stop 2024-02-10T09:32:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 -- 172.21.15.158:0/2572389342 shutdown_connections 2024-02-10T09:32:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 --2- 172.21.15.158:0/2572389342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69700f9150 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:15.375 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 -- 172.21.15.158:0/2572389342 >> 172.21.15.158:0/2572389342 conn(0x7f69700f0b70 msgr2=0x7f69700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 -- 172.21.15.158:0/2572389342 shutdown_connections 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.373+0000 7f6977a1b700 1 -- 172.21.15.158:0/2572389342 wait complete. 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f6977a1b700 1 Processor -- start 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f6977a1b700 1 -- start start 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f6977a1b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69701067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f6977a1b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970106c90 con 0x7f6970060220 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f69757b7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69701067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f69757b7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69701067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:55568/0 (socket says 172.21.15.158:55568) 2024-02-10T09:32:15.376 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f69757b7700 1 -- 172.21.15.158:0/2756815526 learned_addr learned my addr 172.21.15.158:0/2756815526 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.374+0000 7f69757b7700 1 -- 172.21.15.158:0/2756815526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960005040 con 0x7f6970060220 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.375+0000 7f69757b7700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69701067a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f6960009830 tx=0x7f696000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.375+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6960009320 con 0x7f6970060220 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.375+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6970106e90 con 0x7f6970060220 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.375+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6960013380 con 0x7f6970060220 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.375+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6960014420 con 0x7f6970060220 2024-02-10T09:32:15.377 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.375+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6970107320 con 0x7f6970060220 2024-02-10T09:32:15.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.376+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f696001e420 con 0x7f6970060220 2024-02-10T09:32:15.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.376+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6970045ad0 con 0x7f6970060220 2024-02-10T09:32:15.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.376+0000 7f6965ffb700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f695c02ff60 0x7f695c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:15.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.376+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6960040ee0 con 0x7f6970060220 2024-02-10T09:32:15.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.376+0000 7f6974fb6700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f695c02ff60 0x7f695c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:15.378 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.376+0000 7f6974fb6700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f695c02ff60 0x7f695c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f696c0096a0 tx=0x7f696c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:15.380 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.380+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f696001c970 con 0x7f6970060220 2024-02-10T09:32:15.586 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.586+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69700fc090 con 0x7f6970060220 2024-02-10T09:32:15.588 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.587+0000 7f6965ffb700 1 -- 172.21.15.158:0/2756815526 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f696002c2d0 con 0x7f6970060220 2024-02-10T09:32:15.588 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:15.588 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f695c02ff60 msgr2=0x7f695c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:15.589 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f695c02ff60 0x7f695c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f696c0096a0 tx=0x7f696c006b40).stop 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 msgr2=0x7f69701067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69701067a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f6960009830 tx=0x7f696000b040).stop 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 shutdown_connections 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f695c02ff60 0x7f695c032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 --2- 172.21.15.158:0/2756815526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6970060220 0x7f69701067a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 >> 172.21.15.158:0/2756815526 conn(0x7f69700f0b70 msgr2=0x7f69700f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 shutdown_connections 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:15.589+0000 7f6977a1b700 1 -- 172.21.15.158:0/2756815526 wait complete. 2024-02-10T09:32:15.590 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:14.130620+0000 mgr.smithi083.dowdym (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:16.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:15.587537+0000 mon.smithi083 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.158:0/2756815526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:17.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:17.325 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:16.130944+0000 mgr.smithi083.dowdym (mgr.14162) 309 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:19.224 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.222+0000 7f3926321700 1 -- 172.21.15.158:0/2675184378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 msgr2=0x7f39200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:19.224 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.222+0000 7f3926321700 1 --2- 172.21.15.158:0/2675184378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3910009fb0 tx=0x7f3910009b30).stop 2024-02-10T09:32:19.224 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.222+0000 7f3926321700 1 -- 172.21.15.158:0/2675184378 shutdown_connections 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.222+0000 7f3926321700 1 --2- 172.21.15.158:0/2675184378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.222+0000 7f3926321700 1 -- 172.21.15.158:0/2675184378 >> 172.21.15.158:0/2675184378 conn(0x7f39200f09a0 msgr2=0x7f39200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f3926321700 1 -- 172.21.15.158:0/2675184378 shutdown_connections 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f3926321700 1 -- 172.21.15.158:0/2675184378 wait complete. 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f3926321700 1 Processor -- start 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f3926321700 1 -- start start 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f3926321700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f3926321700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39200f13e0 con 0x7f39200f51a0 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f392531f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f392531f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:55582/0 (socket says 172.21.15.158:55582) 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.223+0000 7f392531f700 1 -- 172.21.15.158:0/1150470839 learned_addr learned my addr 172.21.15.158:0/1150470839 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f392531f700 1 -- 172.21.15.158:0/1150470839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3910005040 con 0x7f39200f51a0 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f392531f700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f2f60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f39100089f0 tx=0x7f391000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:19.225 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3910009320 con 0x7f39200f51a0 2024-02-10T09:32:19.226 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39200f1580 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3910013380 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3910014e20 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.224+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39200f1a10 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.225+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39100095f0 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.225+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3920059100 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.225+0000 7f3915ffb700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f390c02ff10 0x7f390c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.226+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3910040f80 con 0x7f39200f51a0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.226+0000 7f3924b1e700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f390c02ff10 0x7f390c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:19.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.226+0000 7f3924b1e700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f390c02ff10 0x7f390c0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f391c0096a0 tx=0x7f391c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:19.230 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.230+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3910014420 con 0x7f39200f51a0 2024-02-10T09:32:19.436 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.436+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3920045ad0 con 0x7f39200f51a0 2024-02-10T09:32:19.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.437+0000 7f3915ffb700 1 -- 172.21.15.158:0/1150470839 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3910010230 con 0x7f39200f51a0 2024-02-10T09:32:19.437 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:19.437 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f390c02ff10 msgr2=0x7f390c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f390c02ff10 0x7f390c0323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f391c0096a0 tx=0x7f391c006b40).stop 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 msgr2=0x7f39200f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f2f60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f39100089f0 tx=0x7f391000b040).stop 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 shutdown_connections 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f390c02ff10 0x7f390c0323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 --2- 172.21.15.158:0/1150470839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39200f51a0 0x7f39200f2f60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 >> 172.21.15.158:0/1150470839 conn(0x7f39200f09a0 msgr2=0x7f392005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 shutdown_connections 2024-02-10T09:32:19.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:19.438+0000 7f3926321700 1 -- 172.21.15.158:0/1150470839 wait complete. 2024-02-10T09:32:19.440 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:18.131384+0000 mgr.smithi083.dowdym (mgr.14162) 310 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:19.437139+0000 mon.smithi083 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.158:0/1150470839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:21.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:21.027 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:22.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:20.131830+0000 mgr.smithi083.dowdym (mgr.14162) 311 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:22.819 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 -- 172.21.15.158:0/1967803941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f40f5410 msgr2=0x7fb8f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:22.819 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 --2- 172.21.15.158:0/1967803941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f40f5410 0x7fb8f40f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fb8e4009fb0 tx=0x7fb8e4009b30).stop 2024-02-10T09:32:22.819 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 -- 172.21.15.158:0/1967803941 shutdown_connections 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 --2- 172.21.15.158:0/1967803941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f40f5410 0x7fb8f40f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 -- 172.21.15.158:0/1967803941 >> 172.21.15.158:0/1967803941 conn(0x7fb8f40f0b70 msgr2=0x7fb8f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 -- 172.21.15.158:0/1967803941 shutdown_connections 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.816+0000 7fb8fbec4700 1 -- 172.21.15.158:0/1967803941 wait complete. 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8fbec4700 1 Processor -- start 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8fbec4700 1 -- start start 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8fbec4700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 0x7fb8f4106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8fbec4700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f4107210 con 0x7fb8f4106900 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8f9c60700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 0x7fb8f4106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8f9c60700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 0x7fb8f4106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:51040/0 (socket says 172.21.15.158:51040) 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.817+0000 7fb8f9c60700 1 -- 172.21.15.158:0/3412663311 learned_addr learned my addr 172.21.15.158:0/3412663311 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8f9c60700 1 -- 172.21.15.158:0/3412663311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4005040 con 0x7fb8f4106900 2024-02-10T09:32:22.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8f9c60700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 0x7fb8f4106d20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fb8e4009f80 tx=0x7fb8e4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4013380 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f4107410 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8e4013790 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4014420 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.818+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f4107f60 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.819+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8e4025060 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.819+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f4059100 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.819+0000 7fb8ea7fc700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb8e002ff60 0x7fb8e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.819+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8e4040d80 con 0x7fb8f4106900 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.819+0000 7fb8f945f700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb8e002ff60 0x7fb8e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:22.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.819+0000 7fb8f945f700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb8e002ff60 0x7fb8e0032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb8f00096a0 tx=0x7fb8f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:22.824 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:22.824+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8e400f280 con 0x7fb8f4106900 2024-02-10T09:32:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.030+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8f41086c0 con 0x7fb8f4106900 2024-02-10T09:32:23.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.030+0000 7fb8ea7fc700 1 -- 172.21.15.158:0/3412663311 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8e402d2d0 con 0x7fb8f4106900 2024-02-10T09:32:23.031 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:23.031 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb8e002ff60 msgr2=0x7fb8e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:23.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb8e002ff60 0x7fb8e0032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb8f00096a0 tx=0x7fb8f0006b40).stop 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 msgr2=0x7fb8f4106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 0x7fb8f4106d20 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fb8e4009f80 tx=0x7fb8e4009b30).stop 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 shutdown_connections 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb8e002ff60 0x7fb8e0032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 --2- 172.21.15.158:0/3412663311 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb8f4106900 0x7fb8f4106d20 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 >> 172.21.15.158:0/3412663311 conn(0x7fb8f40f0b70 msgr2=0x7fb8f405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 shutdown_connections 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:23.032+0000 7fb8fbec4700 1 -- 172.21.15.158:0/3412663311 wait complete. 2024-02-10T09:32:23.033 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:22.132172+0000 mgr.smithi083.dowdym (mgr.14162) 312 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:24.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:23.030687+0000 mon.smithi083 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.158:0/3412663311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:24.768 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:24.769 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:25.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:32:23.818901+0000 mgr.smithi083.dowdym (mgr.14162) 313 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi083 ... 2024-02-10T09:32:25.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-10T09:32:25.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-10T09:32:25.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:30:34Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:25.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:31:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:31:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: main() 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: r = ctx.func(ctx) 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: return func(ctx) 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-10T09:32:25.027 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: raise RuntimeError('uid/gid not found') 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: RuntimeError: uid/gid not found 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: yield (conn, connr) 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: code, '\n'.join(err))) 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi083 ... 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi083 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:30:34Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/prometheus:v2.18.1: reading manifest sha256:b4e6cd0275a26750505e539f8528e891053434ebd3972be02645bed5f02f0795 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:31:08Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:25.028 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr time="2024-02-10T09:31:46Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: Traceback (most recent call last): 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: main() 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: r = ctx.func(ctx) 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: return func(ctx) 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: File "/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: raise RuntimeError('uid/gid not found') 2024-02-10T09:32:25.029 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: RuntimeError: uid/gid not found 2024-02-10T09:32:25.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:23.819453+0000 mon.smithi083 (mon.0) 299 : audit [DBG] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:32:25.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:24.046327+0000 mon.smithi083 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:25.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:24.051318+0000 mon.smithi083 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:25.030 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:24.287850+0000 mon.smithi083 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:26.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:24.132441+0000 mgr.smithi083.dowdym (mgr.14162) 314 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:26.526 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:25.857331+0000 mon.smithi083 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:28.525 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:28 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:26.132705+0000 mgr.smithi083.dowdym (mgr.14162) 315 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:28.569 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 -- 172.21.15.158:0/3778039903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 msgr2=0x7f8a200f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 --2- 172.21.15.158:0/3778039903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f90f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8a10009fb0 tx=0x7f8a10009b30).stop 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 -- 172.21.15.158:0/3778039903 shutdown_connections 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 --2- 172.21.15.158:0/3778039903 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f90f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 -- 172.21.15.158:0/3778039903 >> 172.21.15.158:0/3778039903 conn(0x7f8a200f0b70 msgr2=0x7f8a200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 -- 172.21.15.158:0/3778039903 shutdown_connections 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 -- 172.21.15.158:0/3778039903 wait complete. 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.568+0000 7f8a26381700 1 Processor -- start 2024-02-10T09:32:28.570 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a26381700 1 -- start start 2024-02-10T09:32:28.571 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a26381700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:28.571 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a26381700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200f4ff0 con 0x7f8a20060280 2024-02-10T09:32:28.571 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a1ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a1ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:51054/0 (socket says 172.21.15.158:51054) 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a1ffff700 1 -- 172.21.15.158:0/1847584427 learned_addr learned my addr 172.21.15.158:0/1847584427 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a1ffff700 1 -- 172.21.15.158:0/1847584427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10005040 con 0x7f8a20060280 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a1ffff700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f6b70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8a100089f0 tx=0x7f8a1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.569+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a10009320 con 0x7f8a20060280 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.570+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a200f5190 con 0x7f8a20060280 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.570+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a10013380 con 0x7f8a20060280 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.570+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a10014e20 con 0x7f8a20060280 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.570+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a200f5620 con 0x7f8a20060280 2024-02-10T09:32:28.572 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.570+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a100263a0 con 0x7f8a20060280 2024-02-10T09:32:28.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.571+0000 7f8a1cff9700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8a0802ff10 0x7f8a080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:28.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.571+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a10040eb0 con 0x7f8a20060280 2024-02-10T09:32:28.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.571+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a20059100 con 0x7f8a20060280 2024-02-10T09:32:28.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.571+0000 7f8a1f7fe700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8a0802ff10 0x7f8a080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:28.573 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.571+0000 7f8a1f7fe700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8a0802ff10 0x7f8a080323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8a140096a0 tx=0x7f8a14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:28.577 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.576+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a10012920 con 0x7f8a20060280 2024-02-10T09:32:28.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.794+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a200f5c80 con 0x7f8a20060280 2024-02-10T09:32:28.796 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.794+0000 7f8a1cff9700 1 -- 172.21.15.158:0/1847584427 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a1002b020 con 0x7f8a20060280 2024-02-10T09:32:28.796 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:28.796 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8a0802ff10 msgr2=0x7f8a080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8a0802ff10 0x7f8a080323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8a140096a0 tx=0x7f8a14006b40).stop 2024-02-10T09:32:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 msgr2=0x7f8a200f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:28.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f6b70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f8a100089f0 tx=0x7f8a1000b040).stop 2024-02-10T09:32:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 shutdown_connections 2024-02-10T09:32:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8a0802ff10 0x7f8a080323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 --2- 172.21.15.158:0/1847584427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a20060280 0x7f8a200f6b70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.797+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 >> 172.21.15.158:0/1847584427 conn(0x7f8a200f0b70 msgr2=0x7f8a200f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.798+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 shutdown_connections 2024-02-10T09:32:28.799 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:28.798+0000 7f8a26381700 1 -- 172.21.15.158:0/1847584427 wait complete. 2024-02-10T09:32:28.800 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:28.132954+0000 mgr.smithi083.dowdym (mgr.14162) 316 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:28.579438+0000 mon.smithi083 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:28.796006+0000 mon.smithi083 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.158:0/1847584427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:28.862115+0000 mon.smithi083 (mon.0) 306 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:28.862891+0000 mgr.smithi083.dowdym (mgr.14162) 317 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:28.865809+0000 mon.smithi083 (mon.0) 307 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cephadm 2024-02-10T09:32:28.867925+0000 mgr.smithi083.dowdym (mgr.14162) 318 : cephadm [INF] Deploying daemon alertmanager.smithi083 on smithi083 2024-02-10T09:32:30.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:29.052938+0000 mon.smithi083 (mon.0) 308 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:32:30.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:30.394 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:32.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:30.863246+0000 mgr.smithi083.dowdym (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 -- 172.21.15.158:0/3554359206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 msgr2=0x7fb75c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 --2- 172.21.15.158:0/3554359206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c0f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb744009fb0 tx=0x7fb744009b30).stop 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 -- 172.21.15.158:0/3554359206 shutdown_connections 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 --2- 172.21.15.158:0/3554359206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c0f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 -- 172.21.15.158:0/3554359206 >> 172.21.15.158:0/3554359206 conn(0x7fb75c0f0b70 msgr2=0x7fb75c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 -- 172.21.15.158:0/3554359206 shutdown_connections 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.319+0000 7fb760c11700 1 -- 172.21.15.158:0/3554359206 wait complete. 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb760c11700 1 Processor -- start 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb760c11700 1 -- start start 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb760c11700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:32.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb760c11700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c060220 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb75a59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb75a59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47930/0 (socket says 172.21.15.158:47930) 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb75a59c700 1 -- 172.21.15.158:0/1101402587 learned_addr learned my addr 172.21.15.158:0/1101402587 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb75a59c700 1 -- 172.21.15.158:0/1101402587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744005040 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.320+0000 7fb75a59c700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c109500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb744005010 tx=0x7fb74400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.321+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb744009320 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.321+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c060420 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.321+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb744013380 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.321+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb744014e50 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.321+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c109bb0 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.321+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7440263a0 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.322+0000 7fb752ffd700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb74802ff10 0x7fb7480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.322+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb744041030 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.322+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c059100 con 0x7fb75c0f5410 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.322+0000 7fb759d9b700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb74802ff10 0x7fb7480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:32.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.322+0000 7fb759d9b700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb74802ff10 0x7fb7480323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb74c0096a0 tx=0x7fb74c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:32.348 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.326+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb744012920 con 0x7fb75c0f5410 2024-02-10T09:32:32.532 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.531+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb75c0fafa0 con 0x7fb75c0f5410 2024-02-10T09:32:32.533 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.531+0000 7fb752ffd700 1 -- 172.21.15.158:0/1101402587 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb74402b020 con 0x7fb75c0f5410 2024-02-10T09:32:32.533 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:32.533 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:32.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb74802ff10 msgr2=0x7fb7480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:32.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb74802ff10 0x7fb7480323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb74c0096a0 tx=0x7fb74c006b40).stop 2024-02-10T09:32:32.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 msgr2=0x7fb75c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:32.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c109500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb744005010 tx=0x7fb74400b040).stop 2024-02-10T09:32:32.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 shutdown_connections 2024-02-10T09:32:32.534 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fb74802ff10 0x7fb7480323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:32.535 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 --2- 172.21.15.158:0/1101402587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb75c0f5410 0x7fb75c109500 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:32.535 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 >> 172.21.15.158:0/1101402587 conn(0x7fb75c0f0b70 msgr2=0x7fb75c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:32.535 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 shutdown_connections 2024-02-10T09:32:32.535 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:32.533+0000 7fb760c11700 1 -- 172.21.15.158:0/1101402587 wait complete. 2024-02-10T09:32:32.535 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:33.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:32 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:32.532754+0000 mon.smithi083 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.158:0/1101402587' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:34.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:34.241 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:34.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:32.863562+0000 mgr.smithi083.dowdym (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:36.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:34.863931+0000 mgr.smithi083.dowdym (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/2066632344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 msgr2=0x7f8f780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 --2- 172.21.15.158:0/2066632344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f780f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8f6c009fb0 tx=0x7f8f6c009b30).stop 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/2066632344 shutdown_connections 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 --2- 172.21.15.158:0/2066632344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f780f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/2066632344 >> 172.21.15.158:0/2066632344 conn(0x7f8f780f1c70 msgr2=0x7f8f780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/2066632344 shutdown_connections 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.208+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/2066632344 wait complete. 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.209+0000 7f8f7f0c8700 1 Processor -- start 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.209+0000 7f8f7f0c8700 1 -- start start 2024-02-10T09:32:36.211 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.209+0000 7f8f7f0c8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:36.212 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.209+0000 7f8f7f0c8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78106cd0 con 0x7f8f780f6510 2024-02-10T09:32:36.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7ce64700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:36.213 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7ce64700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:47950/0 (socket says 172.21.15.158:47950) 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7ce64700 1 -- 172.21.15.158:0/3464245064 learned_addr learned my addr 172.21.15.158:0/3464245064 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7ce64700 1 -- 172.21.15.158:0/3464245064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c005040 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7ce64700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f781067e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8f6c005010 tx=0x7f8f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c009320 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f78106ed0 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f6c013380 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c014e50 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.210+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f78107360 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.211+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f6c014420 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.211+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78045ad0 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.211+0000 7f8f757fa700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8f6802ff10 0x7f8f680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.211+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f6c040f40 con 0x7f8f780f6510 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.212+0000 7f8f77fff700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8f6802ff10 0x7f8f680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:36.214 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.212+0000 7f8f77fff700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8f6802ff10 0x7f8f680323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8f640096a0 tx=0x7f8f64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:36.217 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.216+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f6c01dd50 con 0x7f8f780f6510 2024-02-10T09:32:36.422 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.420+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f78108210 con 0x7f8f780f6510 2024-02-10T09:32:36.422 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.421+0000 7f8f757fa700 1 -- 172.21.15.158:0/3464245064 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f6c012330 con 0x7f8f780f6510 2024-02-10T09:32:36.422 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:36.423 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:36.424 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8f6802ff10 msgr2=0x7f8f680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:36.424 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8f6802ff10 0x7f8f680323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f8f640096a0 tx=0x7f8f64006b40).stop 2024-02-10T09:32:36.424 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 msgr2=0x7f8f781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:36.424 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f781067e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f8f6c005010 tx=0x7f8f6c00b040).stop 2024-02-10T09:32:36.424 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 shutdown_connections 2024-02-10T09:32:36.424 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f8f6802ff10 0x7f8f680323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:36.425 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 --2- 172.21.15.158:0/3464245064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f780f6510 0x7f8f781067e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:36.425 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 >> 172.21.15.158:0/3464245064 conn(0x7f8f780f1c70 msgr2=0x7f8f780f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:36.425 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 shutdown_connections 2024-02-10T09:32:36.425 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:36.422+0000 7f8f7f0c8700 1 -- 172.21.15.158:0/3464245064 wait complete. 2024-02-10T09:32:36.425 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:37.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:36 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:36.422330+0000 mon.smithi083 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.158:0/3464245064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:38.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:36.864252+0000 mgr.smithi083.dowdym (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:38.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:38.333 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:40.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:38.864657+0000 mgr.smithi083.dowdym (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:40.395 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/1364713489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 msgr2=0x7f6e680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:40.395 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 --2- 172.21.15.158:0/1364713489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6e5c009fb0 tx=0x7f6e5c009b30).stop 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/1364713489 shutdown_connections 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 --2- 172.21.15.158:0/1364713489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/1364713489 >> 172.21.15.158:0/1364713489 conn(0x7f6e680f0b70 msgr2=0x7f6e680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/1364713489 shutdown_connections 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.393+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/1364713489 wait complete. 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.394+0000 7f6e6ec6a700 1 Processor -- start 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.394+0000 7f6e6ec6a700 1 -- start start 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.394+0000 7f6e6ec6a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:40.396 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.394+0000 7f6e6ec6a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e680fb960 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.394+0000 7f6e6ca06700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e6ca06700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52484/0 (socket says 172.21.15.158:52484) 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e6ca06700 1 -- 172.21.15.158:0/56094435 learned_addr learned my addr 172.21.15.158:0/56094435 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e6ca06700 1 -- 172.21.15.158:0/56094435 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c005040 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e6ca06700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680fd4e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6e5c005010 tx=0x7f6e5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e5c009320 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e680fbb00 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e5c013380 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e5c014e50 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.395+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e680fbf90 con 0x7f6e680f5410 2024-02-10T09:32:40.398 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.396+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e5c014420 con 0x7f6e680f5410 2024-02-10T09:32:40.399 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.396+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e68059100 con 0x7f6e680f5410 2024-02-10T09:32:40.399 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.396+0000 7f6e657fa700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6e5802ff10 0x7f6e580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:40.399 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.396+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e5c040f40 con 0x7f6e680f5410 2024-02-10T09:32:40.399 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.396+0000 7f6e67fff700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6e5802ff10 0x7f6e580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:40.399 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.397+0000 7f6e67fff700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6e5802ff10 0x7f6e580323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6e540096a0 tx=0x7f6e54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:40.402 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.401+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e5c0095f0 con 0x7f6e680f5410 2024-02-10T09:32:40.610 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.608+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e680fafa0 con 0x7f6e680f5410 2024-02-10T09:32:40.610 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.609+0000 7f6e657fa700 1 -- 172.21.15.158:0/56094435 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e5c01db70 con 0x7f6e680f5410 2024-02-10T09:32:40.610 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:40.611 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:40.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6e5802ff10 msgr2=0x7f6e580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:40.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6e5802ff10 0x7f6e580323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6e540096a0 tx=0x7f6e54006b40).stop 2024-02-10T09:32:40.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 msgr2=0x7f6e680fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:40.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680fd4e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6e5c005010 tx=0x7f6e5c00b040).stop 2024-02-10T09:32:40.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 shutdown_connections 2024-02-10T09:32:40.612 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6e5802ff10 0x7f6e580323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:40.613 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 --2- 172.21.15.158:0/56094435 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6e680f5410 0x7f6e680fd4e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:40.613 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 >> 172.21.15.158:0/56094435 conn(0x7f6e680f0b70 msgr2=0x7f6e6805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:40.613 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 shutdown_connections 2024-02-10T09:32:40.613 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:40.611+0000 7f6e6ec6a700 1 -- 172.21.15.158:0/56094435 wait complete. 2024-02-10T09:32:40.613 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:41.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:40 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:40.610512+0000 mon.smithi083 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.158:0/56094435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:42.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:40.865024+0000 mgr.smithi083.dowdym (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:42.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:42.301 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:44.131 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 -- 172.21.15.158:0/1383108891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 msgr2=0x7f096c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 --2- 172.21.15.158:0/1383108891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c0f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f095c009fb0 tx=0x7f095c009b30).stop 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 -- 172.21.15.158:0/1383108891 shutdown_connections 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 --2- 172.21.15.158:0/1383108891 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c0f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 -- 172.21.15.158:0/1383108891 >> 172.21.15.158:0/1383108891 conn(0x7f096c0f0b70 msgr2=0x7f096c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 -- 172.21.15.158:0/1383108891 shutdown_connections 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.128+0000 7f0973dd5700 1 -- 172.21.15.158:0/1383108891 wait complete. 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.129+0000 7f0973dd5700 1 Processor -- start 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.129+0000 7f0973dd5700 1 -- start start 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.129+0000 7f0973dd5700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.129+0000 7f0973dd5700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c060240 con 0x7f096c0f5410 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.129+0000 7f0971b71700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:44.132 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f0971b71700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52502/0 (socket says 172.21.15.158:52502) 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f0971b71700 1 -- 172.21.15.158:0/3562282118 learned_addr learned my addr 172.21.15.158:0/3562282118 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f0971b71700 1 -- 172.21.15.158:0/3562282118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c005040 con 0x7f096c0f5410 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f0971b71700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c1094e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f095c005010 tx=0x7f095c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f095c009320 con 0x7f096c0f5410 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c060440 con 0x7f096c0f5410 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c109b90 con 0x7f096c0f5410 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f095c013380 con 0x7f096c0f5410 2024-02-10T09:32:44.133 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.130+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f095c014df0 con 0x7f096c0f5410 2024-02-10T09:32:44.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.131+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f095c014420 con 0x7f096c0f5410 2024-02-10T09:32:44.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.131+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c045ad0 con 0x7f096c0f5410 2024-02-10T09:32:44.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.131+0000 7f09627fc700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f095802ff10 0x7f09580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:44.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.131+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f095c0410e0 con 0x7f096c0f5410 2024-02-10T09:32:44.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.132+0000 7f0971370700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f095802ff10 0x7f09580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:44.134 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.132+0000 7f0971370700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f095802ff10 0x7f09580323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f09680096a0 tx=0x7f0968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:44.137 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.136+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f095c00fd10 con 0x7f096c0f5410 2024-02-10T09:32:44.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:42.865437+0000 mgr.smithi083.dowdym (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.343+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f096c10a3b0 con 0x7f096c0f5410 2024-02-10T09:32:44.344 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.343+0000 7f09627fc700 1 -- 172.21.15.158:0/3562282118 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f095c00fb30 con 0x7f096c0f5410 2024-02-10T09:32:44.345 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:44.345 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:44.346 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f095802ff10 msgr2=0x7f09580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f095802ff10 0x7f09580323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f09680096a0 tx=0x7f0968006b40).stop 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 msgr2=0x7f096c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c1094e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f095c005010 tx=0x7f095c00b040).stop 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 shutdown_connections 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f095802ff10 0x7f09580323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 --2- 172.21.15.158:0/3562282118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f096c0f5410 0x7f096c1094e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 >> 172.21.15.158:0/3562282118 conn(0x7f096c0f0b70 msgr2=0x7f096c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 shutdown_connections 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:44.345+0000 7f0973dd5700 1 -- 172.21.15.158:0/3562282118 wait complete. 2024-02-10T09:32:44.347 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:45.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:44 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:44.344774+0000 mon.smithi083 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.158:0/3562282118' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:45.988 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:45.989 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:46.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:44.865829+0000 mgr.smithi083.dowdym (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2299627101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 msgr2=0x7fc6a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 --2- 172.21.15.158:0/2299627101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a80f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fc698009fb0 tx=0x7fc698009b30).stop 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2299627101 shutdown_connections 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 --2- 172.21.15.158:0/2299627101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a80f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2299627101 >> 172.21.15.158:0/2299627101 conn(0x7fc6a80f0b70 msgr2=0x7fc6a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2299627101 shutdown_connections 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.844+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2299627101 wait complete. 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.845+0000 7fc6ae7e2700 1 Processor -- start 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.845+0000 7fc6ae7e2700 1 -- start start 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.845+0000 7fc6ae7e2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.845+0000 7fc6ae7e2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a8106cc0 con 0x7fc6a80f5410 2024-02-10T09:32:47.854 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.845+0000 7fc6a7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.845+0000 7fc6a7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:52506/0 (socket says 172.21.15.158:52506) 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6a7fff700 1 -- 172.21.15.158:0/2674595085 learned_addr learned my addr 172.21.15.158:0/2674595085 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6a7fff700 1 -- 172.21.15.158:0/2674595085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc698005040 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6a7fff700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a81067d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fc698005010 tx=0x7fc69800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc698009320 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a8106ec0 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a8107350 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc698013380 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.846+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc698014df0 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.847+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc69801e420 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.847+0000 7fc6a4ff9700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc69002ff10 0x7fc6900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.847+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc698040eb0 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.847+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a8045ad0 con 0x7fc6a80f5410 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.847+0000 7fc6a77fe700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc69002ff10 0x7fc6900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:47.855 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.848+0000 7fc6a77fe700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc69002ff10 0x7fc6900323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc69c0096a0 tx=0x7fc69c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:47.856 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:47.852+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6980263a0 con 0x7fc6a80f5410 2024-02-10T09:32:48.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.056+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6a81080d0 con 0x7fc6a80f5410 2024-02-10T09:32:48.058 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.057+0000 7fc6a4ff9700 1 -- 172.21.15.158:0/2674595085 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc69802b030 con 0x7fc6a80f5410 2024-02-10T09:32:48.059 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:48.059 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc69002ff10 msgr2=0x7fc6900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc69002ff10 0x7fc6900323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fc69c0096a0 tx=0x7fc69c006b40).stop 2024-02-10T09:32:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 msgr2=0x7fc6a81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a81067d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fc698005010 tx=0x7fc69800b040).stop 2024-02-10T09:32:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 shutdown_connections 2024-02-10T09:32:48.060 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fc69002ff10 0x7fc6900323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 --2- 172.21.15.158:0/2674595085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc6a80f5410 0x7fc6a81067d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 >> 172.21.15.158:0/2674595085 conn(0x7fc6a80f0b70 msgr2=0x7fc6a805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 shutdown_connections 2024-02-10T09:32:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:48.059+0000 7fc6ae7e2700 1 -- 172.21.15.158:0/2674595085 wait complete. 2024-02-10T09:32:48.061 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:48.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:46.866099+0000 mgr.smithi083.dowdym (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:49.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:48 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:48.058508+0000 mon.smithi083 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.158:0/2674595085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:49.676 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:49.676 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:48.866457+0000 mgr.smithi083.dowdym (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 -- 172.21.15.158:0/2663783873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 msgr2=0x7f42e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 --2- 172.21.15.158:0/2663783873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e40f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f42d4009fb0 tx=0x7f42d4009b30).stop 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 -- 172.21.15.158:0/2663783873 shutdown_connections 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 --2- 172.21.15.158:0/2663783873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e40f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 -- 172.21.15.158:0/2663783873 >> 172.21.15.158:0/2663783873 conn(0x7f42e40f1c70 msgr2=0x7f42e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 -- 172.21.15.158:0/2663783873 shutdown_connections 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 -- 172.21.15.158:0/2663783873 wait complete. 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 Processor -- start 2024-02-10T09:32:51.545 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.543+0000 7f42e9501700 1 -- start start 2024-02-10T09:32:51.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e9501700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:51.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e9501700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4069e20 con 0x7f42e40f6510 2024-02-10T09:32:51.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:51.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40574/0 (socket says 172.21.15.158:40574) 2024-02-10T09:32:51.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e2ffd700 1 -- 172.21.15.158:0/1753509245 learned_addr learned my addr 172.21.15.158:0/1753509245 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:51.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e2ffd700 1 -- 172.21.15.158:0/1753509245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d4005040 con 0x7f42e40f6510 2024-02-10T09:32:51.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e2ffd700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e406b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f42d4005010 tx=0x7f42d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:51.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d4009320 con 0x7f42e40f6510 2024-02-10T09:32:51.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e4069fc0 con 0x7f42e40f6510 2024-02-10T09:32:51.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42d4013380 con 0x7f42e40f6510 2024-02-10T09:32:51.548 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d4014e50 con 0x7f42e40f6510 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.544+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e406a450 con 0x7f42e40f6510 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.545+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42d40095f0 con 0x7f42e40f6510 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.545+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e4059100 con 0x7f42e40f6510 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.545+0000 7f42cb7fe700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f42cc02ff10 0x7f42cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.545+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42d4040fa0 con 0x7f42e40f6510 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.546+0000 7f42e27fc700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f42cc02ff10 0x7f42cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:51.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.546+0000 7f42e27fc700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f42cc02ff10 0x7f42cc0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f42d80096a0 tx=0x7f42d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:51.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.550+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42d4014420 con 0x7f42e40f6510 2024-02-10T09:32:51.756 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.755+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42e40fc0a0 con 0x7f42e40f6510 2024-02-10T09:32:51.757 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.755+0000 7f42cb7fe700 1 -- 172.21.15.158:0/1753509245 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42d4010230 con 0x7f42e40f6510 2024-02-10T09:32:51.757 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:51.757 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:51.758 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f42cc02ff10 msgr2=0x7f42cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f42cc02ff10 0x7f42cc0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f42d80096a0 tx=0x7f42d8006b40).stop 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 msgr2=0x7f42e406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e406b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f42d4005010 tx=0x7f42d400b040).stop 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 shutdown_connections 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f42cc02ff10 0x7f42cc0323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 --2- 172.21.15.158:0/1753509245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f42e40f6510 0x7f42e406b9a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 >> 172.21.15.158:0/1753509245 conn(0x7f42e40f1c70 msgr2=0x7f42e40f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 shutdown_connections 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:51.757+0000 7f42e9501700 1 -- 172.21.15.158:0/1753509245 wait complete. 2024-02-10T09:32:51.759 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:52.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:50.866878+0000 mgr.smithi083.dowdym (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:52.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:51.756935+0000 mon.smithi083 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.158:0/1753509245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:53.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:53.558 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:54.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:52.867258+0000 mgr.smithi083.dowdym (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 -- 172.21.15.158:0/4153682989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 msgr2=0x7f6c8c0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 --2- 172.21.15.158:0/4153682989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c0f7d00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f6c7c009fb0 tx=0x7f6c7c009b30).stop 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 -- 172.21.15.158:0/4153682989 shutdown_connections 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 --2- 172.21.15.158:0/4153682989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c0f7d00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 -- 172.21.15.158:0/4153682989 >> 172.21.15.158:0/4153682989 conn(0x7f6c8c0f0120 msgr2=0x7f6c8c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 -- 172.21.15.158:0/4153682989 shutdown_connections 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 -- 172.21.15.158:0/4153682989 wait complete. 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.453+0000 7f6c9089c700 1 Processor -- start 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c9089c700 1 -- start start 2024-02-10T09:32:55.455 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c9089c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c1065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:55.456 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c9089c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c106ad0 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.457 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c8b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c1065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:55.457 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c8b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c1065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40594/0 (socket says 172.21.15.158:40594) 2024-02-10T09:32:55.457 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/283703618 learned_addr learned my addr 172.21.15.158:0/283703618 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:55.457 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c8b7fe700 1 -- 172.21.15.158:0/283703618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c005040 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c8b7fe700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c1065e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6c7c0089f0 tx=0x7f6c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.454+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7c009320 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.455+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c106cd0 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.455+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c7c013380 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.455+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7c014e20 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.455+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c107160 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.455+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c045ad0 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.456+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c7c0095f0 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.456+0000 7f6c73fff700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c7402ff10 0x7f6c740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.456+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c7c040f10 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.456+0000 7f6c8affd700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c7402ff10 0x7f6c740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:55.458 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.456+0000 7f6c8affd700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c7402ff10 0x7f6c740323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6c800096a0 tx=0x7f6c80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:55.461 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.460+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c7c014420 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.668 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.667+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c8c108010 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.669 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.668+0000 7f6c73fff700 1 -- 172.21.15.158:0/283703618 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6c7c01d9d0 con 0x7f6c8c0f78e0 2024-02-10T09:32:55.669 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:55.669 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c7402ff10 msgr2=0x7f6c740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c7402ff10 0x7f6c740323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6c800096a0 tx=0x7f6c80006b40).stop 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 msgr2=0x7f6c8c1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c1065e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6c7c0089f0 tx=0x7f6c7c00b040).stop 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 shutdown_connections 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f6c7402ff10 0x7f6c740323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 --2- 172.21.15.158:0/283703618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c8c0f78e0 0x7f6c8c1065e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 >> 172.21.15.158:0/283703618 conn(0x7f6c8c0f0120 msgr2=0x7f6c8c0f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 shutdown_connections 2024-02-10T09:32:55.671 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:55.669+0000 7f6c9089c700 1 -- 172.21.15.158:0/283703618 wait complete. 2024-02-10T09:32:55.672 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:32:56.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:54.867693+0000 mgr.smithi083.dowdym (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:56.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:55.669177+0000 mon.smithi083 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.158:0/283703618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:32:57.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:32:57.356 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:32:58.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:56.868005+0000 mgr.smithi083.dowdym (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 -- 172.21.15.158:0/3531228332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 msgr2=0x7f576c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 --2- 172.21.15.158:0/3531228332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c0f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f575c009fb0 tx=0x7f575c009b30).stop 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 -- 172.21.15.158:0/3531228332 shutdown_connections 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 --2- 172.21.15.158:0/3531228332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c0f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 -- 172.21.15.158:0/3531228332 >> 172.21.15.158:0/3531228332 conn(0x7f576c0f09a0 msgr2=0x7f576c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 -- 172.21.15.158:0/3531228332 shutdown_connections 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 -- 172.21.15.158:0/3531228332 wait complete. 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 Processor -- start 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.186+0000 7f576b59e700 1 -- start start 2024-02-10T09:32:59.190 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576b59e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576b59e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576c060280 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576a59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576a59c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40620/0 (socket says 172.21.15.158:40620) 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576a59c700 1 -- 172.21.15.158:0/453140385 learned_addr learned my addr 172.21.15.158:0/453140385 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576a59c700 1 -- 172.21.15.158:0/453140385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c005040 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576a59c700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c109290 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f575c0089f0 tx=0x7f575c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575c009320 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f575c013380 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576c060480 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575c014e20 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.187+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576c1099c0 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.188+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576c045ad0 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.188+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f575c0095f0 con 0x7f576c0f51a0 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.188+0000 7f575affd700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f575402ff10 0x7f57540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:32:59.191 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.188+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f575c040f50 con 0x7f576c0f51a0 2024-02-10T09:32:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.189+0000 7f5769d9b700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f575402ff10 0x7f57540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:32:59.192 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.189+0000 7f5769d9b700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f575402ff10 0x7f57540323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f57600096a0 tx=0x7f5760006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:32:59.194 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.193+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f575c014420 con 0x7f576c0f51a0 2024-02-10T09:32:59.400 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.399+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f576c0fad30 con 0x7f576c0f51a0 2024-02-10T09:32:59.401 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.400+0000 7f575affd700 1 -- 172.21.15.158:0/453140385 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f575c01d980 con 0x7f576c0f51a0 2024-02-10T09:32:59.401 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:32:59.402 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f575402ff10 msgr2=0x7f57540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f575402ff10 0x7f57540323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f57600096a0 tx=0x7f5760006b40).stop 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 msgr2=0x7f576c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c109290 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f575c0089f0 tx=0x7f575c00b040).stop 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 shutdown_connections 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f575402ff10 0x7f57540323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 --2- 172.21.15.158:0/453140385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f576c0f51a0 0x7f576c109290 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 >> 172.21.15.158:0/453140385 conn(0x7f576c0f09a0 msgr2=0x7f576c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 shutdown_connections 2024-02-10T09:32:59.403 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:32:59.401+0000 7f576b59e700 1 -- 172.21.15.158:0/453140385 wait complete. 2024-02-10T09:32:59.404 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:32:58.868390+0000 mgr.smithi083.dowdym (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:32:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:32:59.401234+0000 mon.smithi083 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.158:0/453140385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:01.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:01.014 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:02.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:00.868843+0000 mgr.smithi083.dowdym (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:02.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 -- 172.21.15.158:0/641045936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 msgr2=0x7f84e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 --2- 172.21.15.158:0/641045936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e80f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f84dc009fb0 tx=0x7f84dc009b30).stop 2024-02-10T09:33:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 -- 172.21.15.158:0/641045936 shutdown_connections 2024-02-10T09:33:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 --2- 172.21.15.158:0/641045936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e80f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 -- 172.21.15.158:0/641045936 >> 172.21.15.158:0/641045936 conn(0x7f84e80f0b70 msgr2=0x7f84e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 -- 172.21.15.158:0/641045936 shutdown_connections 2024-02-10T09:33:02.871 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.869+0000 7f84ef035700 1 -- 172.21.15.158:0/641045936 wait complete. 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.870+0000 7f84ef035700 1 Processor -- start 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.870+0000 7f84ef035700 1 -- start start 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.870+0000 7f84ef035700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.870+0000 7f84ef035700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8106cc0 con 0x7f84e80f5410 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ecdd1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ecdd1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:32806/0 (socket says 172.21.15.158:32806) 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ecdd1700 1 -- 172.21.15.158:0/1602625577 learned_addr learned my addr 172.21.15.158:0/1602625577 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ecdd1700 1 -- 172.21.15.158:0/1602625577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc005040 con 0x7f84e80f5410 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ecdd1700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e81067d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f84dc0089f0 tx=0x7f84dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:02.873 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84dc009320 con 0x7f84e80f5410 2024-02-10T09:33:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84dc013380 con 0x7f84e80f5410 2024-02-10T09:33:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84dc01ed80 con 0x7f84e80f5410 2024-02-10T09:33:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e8106ec0 con 0x7f84e80f5410 2024-02-10T09:33:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.871+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8107350 con 0x7f84e80f5410 2024-02-10T09:33:02.874 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.872+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84dc014420 con 0x7f84e80f5410 2024-02-10T09:33:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.872+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e8045ad0 con 0x7f84e80f5410 2024-02-10T09:33:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.872+0000 7f84e57fa700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84d802ffb0 0x7f84d8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.872+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84dc041170 con 0x7f84e80f5410 2024-02-10T09:33:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.872+0000 7f84e7fff700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84d802ffb0 0x7f84d8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:02.875 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.873+0000 7f84e7fff700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84d802ffb0 0x7f84d8032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f84d40096a0 tx=0x7f84d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:02.878 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:02.877+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84dc00faf0 con 0x7f84e80f5410 2024-02-10T09:33:03.086 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.085+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84e8108200 con 0x7f84e80f5410 2024-02-10T09:33:03.087 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.086+0000 7f84e57fa700 1 -- 172.21.15.158:0/1602625577 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84dc00faf0 con 0x7f84e80f5410 2024-02-10T09:33:03.087 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:03.087 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84d802ffb0 msgr2=0x7f84d8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84d802ffb0 0x7f84d8032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f84d40096a0 tx=0x7f84d4006b40).stop 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 msgr2=0x7f84e81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e81067d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f84dc0089f0 tx=0x7f84dc00b040).stop 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 shutdown_connections 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f84d802ffb0 0x7f84d8032470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 --2- 172.21.15.158:0/1602625577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f84e80f5410 0x7f84e81067d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.087+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 >> 172.21.15.158:0/1602625577 conn(0x7f84e80f0b70 msgr2=0x7f84e805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:03.089 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.088+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 shutdown_connections 2024-02-10T09:33:03.090 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:03.088+0000 7f84ef035700 1 -- 172.21.15.158:0/1602625577 wait complete. 2024-02-10T09:33:03.090 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:04.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:02.869280+0000 mgr.smithi083.dowdym (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:04.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:03.087253+0000 mon.smithi083 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.158:0/1602625577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:04.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:04.776 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:06.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:04.869679+0000 mgr.smithi083.dowdym (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:06.654 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.651+0000 7f0788a2b700 1 -- 172.21.15.158:0/1818759895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 msgr2=0x7f07840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:06.654 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.651+0000 7f0788a2b700 1 --2- 172.21.15.158:0/1818759895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f07840f55c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f0774009fb0 tx=0x7f0774009b30).stop 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.651+0000 7f0788a2b700 1 -- 172.21.15.158:0/1818759895 shutdown_connections 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.651+0000 7f0788a2b700 1 --2- 172.21.15.158:0/1818759895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f07840f55c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.651+0000 7f0788a2b700 1 -- 172.21.15.158:0/1818759895 >> 172.21.15.158:0/1818759895 conn(0x7f07840f09a0 msgr2=0x7f07840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.652+0000 7f0788a2b700 1 -- 172.21.15.158:0/1818759895 shutdown_connections 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.652+0000 7f0788a2b700 1 -- 172.21.15.158:0/1818759895 wait complete. 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.652+0000 7f0788a2b700 1 Processor -- start 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.652+0000 7f0788a2b700 1 -- start start 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.652+0000 7f0788a2b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f078410b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.652+0000 7f0788a2b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07841070c0 con 0x7f07840f51a0 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0782d9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f078410b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0782d9d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f078410b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:32820/0 (socket says 172.21.15.158:32820) 2024-02-10T09:33:06.655 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0782d9d700 1 -- 172.21.15.158:0/360009638 learned_addr learned my addr 172.21.15.158:0/360009638 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0782d9d700 1 -- 172.21.15.158:0/360009638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774005040 con 0x7f07840f51a0 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0782d9d700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f078410b6d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f07740089f0 tx=0x7f077400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0774009320 con 0x7f07840f51a0 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0774013380 con 0x7f07840f51a0 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0784060240 con 0x7f07840f51a0 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0774014e20 con 0x7f07840f51a0 2024-02-10T09:33:06.656 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.653+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078410bbc0 con 0x7f07840f51a0 2024-02-10T09:33:06.657 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.654+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0774014420 con 0x7f07840f51a0 2024-02-10T09:33:06.657 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.654+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0784045ad0 con 0x7f07840f51a0 2024-02-10T09:33:06.657 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.654+0000 7f076b7fe700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f076c02ff10 0x7f076c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:06.657 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.654+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0774040ff0 con 0x7f07840f51a0 2024-02-10T09:33:06.657 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.654+0000 7f078259c700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f076c02ff10 0x7f076c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:06.657 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.655+0000 7f078259c700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f076c02ff10 0x7f076c0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f07780096a0 tx=0x7f0778006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:06.660 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.659+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07740095f0 con 0x7f07840f51a0 2024-02-10T09:33:06.868 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.866+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f078410c3e0 con 0x7f07840f51a0 2024-02-10T09:33:06.868 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.867+0000 7f076b7fe700 1 -- 172.21.15.158:0/360009638 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f077401db70 con 0x7f07840f51a0 2024-02-10T09:33:06.868 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:06.868 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f076c02ff10 msgr2=0x7f076c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f076c02ff10 0x7f076c0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f07780096a0 tx=0x7f0778006b40).stop 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 msgr2=0x7f078410b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f078410b6d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f07740089f0 tx=0x7f077400b040).stop 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 shutdown_connections 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f076c02ff10 0x7f076c0323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 --2- 172.21.15.158:0/360009638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07840f51a0 0x7f078410b6d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 >> 172.21.15.158:0/360009638 conn(0x7f07840f09a0 msgr2=0x7f07840f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 shutdown_connections 2024-02-10T09:33:06.870 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:06.868+0000 7f0788a2b700 1 -- 172.21.15.158:0/360009638 wait complete. 2024-02-10T09:33:06.871 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:07.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:06 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:06.868248+0000 mon.smithi083 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.158:0/360009638' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:08.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:06.869935+0000 mgr.smithi083.dowdym (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:08.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:08.510 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:10.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:08.870325+0000 mgr.smithi083.dowdym (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:10.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 -- 172.21.15.158:0/3111540020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a00060280 msgr2=0x7f5a000f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:10.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 --2- 172.21.15.158:0/3111540020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a00060280 0x7f5a000f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f59f0009fb0 tx=0x7f59f0009b30).stop 2024-02-10T09:33:10.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 -- 172.21.15.158:0/3111540020 shutdown_connections 2024-02-10T09:33:10.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 --2- 172.21.15.158:0/3111540020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a00060280 0x7f5a000f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:10.550 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 -- 172.21.15.158:0/3111540020 >> 172.21.15.158:0/3111540020 conn(0x7f5a000f0b70 msgr2=0x7f5a000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:10.550 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 -- 172.21.15.158:0/3111540020 shutdown_connections 2024-02-10T09:33:10.550 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.547+0000 7f5a0707f700 1 -- 172.21.15.158:0/3111540020 wait complete. 2024-02-10T09:33:10.550 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.548+0000 7f5a0707f700 1 Processor -- start 2024-02-10T09:33:10.550 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.548+0000 7f5a0707f700 1 -- start start 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.548+0000 7f5a0707f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 0x7f5a00106cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.548+0000 7f5a0707f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a001071b0 con 0x7f5a001068a0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.548+0000 7f5a04e1b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 0x7f5a00106cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f5a04e1b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 0x7f5a00106cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40832/0 (socket says 172.21.15.158:40832) 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f5a04e1b700 1 -- 172.21.15.158:0/2349485504 learned_addr learned my addr 172.21.15.158:0/2349485504 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f5a04e1b700 1 -- 172.21.15.158:0/2349485504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f0005040 con 0x7f5a001068a0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f5a04e1b700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 0x7f5a00106cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f59f0011000 tx=0x7f59f0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59f0013380 con 0x7f5a001068a0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a001073b0 con 0x7f5a001068a0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59f0013790 con 0x7f5a001068a0 2024-02-10T09:33:10.552 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59f001ee70 con 0x7f5a001068a0 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.549+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a00107f00 con 0x7f5a001068a0 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.550+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59f0025060 con 0x7f5a001068a0 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.550+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a00059100 con 0x7f5a001068a0 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.550+0000 7f59fd7fa700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59e802ff60 0x7f59e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.550+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59f0040f10 con 0x7f5a001068a0 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.550+0000 7f59fffff700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59e802ff60 0x7f59e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:10.553 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.551+0000 7f59fffff700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59e802ff60 0x7f59e8032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f59f40096a0 tx=0x7f59f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:10.556 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.555+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59f001c950 con 0x7f5a001068a0 2024-02-10T09:33:10.761 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.760+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a00045ad0 con 0x7f5a001068a0 2024-02-10T09:33:10.762 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.761+0000 7f59fd7fa700 1 -- 172.21.15.158:0/2349485504 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59f002d2d0 con 0x7f5a001068a0 2024-02-10T09:33:10.762 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:10.762 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:10.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.762+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59e802ff60 msgr2=0x7f59e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:10.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.762+0000 7f5a0707f700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59e802ff60 0x7f59e8032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f59f40096a0 tx=0x7f59f4006b40).stop 2024-02-10T09:33:10.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.762+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 msgr2=0x7f5a00106cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:10.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.762+0000 7f5a0707f700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 0x7f5a00106cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f59f0011000 tx=0x7f59f0009b30).stop 2024-02-10T09:33:10.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.763+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 shutdown_connections 2024-02-10T09:33:10.764 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.763+0000 7f5a0707f700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f59e802ff60 0x7f59e8032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:10.765 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.763+0000 7f5a0707f700 1 --2- 172.21.15.158:0/2349485504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a001068a0 0x7f5a00106cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:10.765 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.763+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 >> 172.21.15.158:0/2349485504 conn(0x7f5a000f0b70 msgr2=0x7f5a000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:10.765 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.763+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 shutdown_connections 2024-02-10T09:33:10.765 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:10.763+0000 7f5a0707f700 1 -- 172.21.15.158:0/2349485504 wait complete. 2024-02-10T09:33:10.765 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:11.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:10 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:10.762332+0000 mon.smithi083 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.158:0/2349485504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:10.870664+0000 mgr.smithi083.dowdym (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:12.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:12.483 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:14.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:12.871024+0000 mgr.smithi083.dowdym (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.436+0000 7ff93653a700 1 -- 172.21.15.158:0/189993334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300f51a0 msgr2=0x7ff9300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.436+0000 7ff93653a700 1 --2- 172.21.15.158:0/189993334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300f51a0 0x7ff9300f55c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ff920009fb0 tx=0x7ff920009b30).stop 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 -- 172.21.15.158:0/189993334 shutdown_connections 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 --2- 172.21.15.158:0/189993334 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300f51a0 0x7ff9300f55c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 -- 172.21.15.158:0/189993334 >> 172.21.15.158:0/189993334 conn(0x7ff9300f09a0 msgr2=0x7ff9300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 -- 172.21.15.158:0/189993334 shutdown_connections 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 -- 172.21.15.158:0/189993334 wait complete. 2024-02-10T09:33:14.441 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 Processor -- start 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 -- start start 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 0x7ff930109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.437+0000 7ff93653a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff930109920 con 0x7ff9300601e0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff935538700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 0x7ff930109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff935538700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 0x7ff930109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40852/0 (socket says 172.21.15.158:40852) 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff935538700 1 -- 172.21.15.158:0/1661087306 learned_addr learned my addr 172.21.15.158:0/1661087306 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff935538700 1 -- 172.21.15.158:0/1661087306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff920005040 con 0x7ff9300601e0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff935538700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 0x7ff930109430 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ff920011000 tx=0x7ff920009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff920013380 con 0x7ff9300601e0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff930109a60 con 0x7ff9300601e0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff920013790 con 0x7ff9300601e0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff92001ee70 con 0x7ff9300601e0 2024-02-10T09:33:14.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.438+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff930109e90 con 0x7ff9300601e0 2024-02-10T09:33:14.443 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.439+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff920025060 con 0x7ff9300601e0 2024-02-10T09:33:14.443 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.439+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff930059100 con 0x7ff9300601e0 2024-02-10T09:33:14.443 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.439+0000 7ff925ffb700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff91c02ff60 0x7ff91c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:14.443 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.439+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff920040ea0 con 0x7ff9300601e0 2024-02-10T09:33:14.443 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.440+0000 7ff934d37700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff91c02ff60 0x7ff91c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:14.444 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.440+0000 7ff934d37700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff91c02ff60 0x7ff91c032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7ff92c0096a0 tx=0x7ff92c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:14.445 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.444+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff92001c6c0 con 0x7ff9300601e0 2024-02-10T09:33:14.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.649+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff930045ad0 con 0x7ff9300601e0 2024-02-10T09:33:14.651 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.649+0000 7ff925ffb700 1 -- 172.21.15.158:0/1661087306 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff92002d2d0 con 0x7ff9300601e0 2024-02-10T09:33:14.651 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:14.651 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff91c02ff60 msgr2=0x7ff91c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff91c02ff60 0x7ff91c032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7ff92c0096a0 tx=0x7ff92c006b40).stop 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 msgr2=0x7ff930109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 0x7ff930109430 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ff920011000 tx=0x7ff920009b30).stop 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 shutdown_connections 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7ff91c02ff60 0x7ff91c032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 --2- 172.21.15.158:0/1661087306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff9300601e0 0x7ff930109430 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 >> 172.21.15.158:0/1661087306 conn(0x7ff9300f09a0 msgr2=0x7ff9300f14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 shutdown_connections 2024-02-10T09:33:14.653 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:14.651+0000 7ff93653a700 1 -- 172.21.15.158:0/1661087306 wait complete. 2024-02-10T09:33:14.654 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:15.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:14.177726+0000 mon.smithi083 (mon.0) 320 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:33:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:14.184764+0000 mon.smithi083 (mon.0) 321 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:33:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:14.650943+0000 mon.smithi083 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.158:0/1661087306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:16.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:14.871282+0000 mgr.smithi083.dowdym (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:16.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:16.351 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:18.201 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 -- 172.21.15.158:0/2697310054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 msgr2=0x7fcfb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 --2- 172.21.15.158:0/2697310054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb40f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fcfa4009fb0 tx=0x7fcfa4009b30).stop 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 -- 172.21.15.158:0/2697310054 shutdown_connections 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 --2- 172.21.15.158:0/2697310054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb40f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 -- 172.21.15.158:0/2697310054 >> 172.21.15.158:0/2697310054 conn(0x7fcfb40f0b70 msgr2=0x7fcfb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 -- 172.21.15.158:0/2697310054 shutdown_connections 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.199+0000 7fcfbc434700 1 -- 172.21.15.158:0/2697310054 wait complete. 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.200+0000 7fcfbc434700 1 Processor -- start 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.200+0000 7fcfbc434700 1 -- start start 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.200+0000 7fcfbc434700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:18.202 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.200+0000 7fcfbc434700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4106ca0 con 0x7fcfb40f5410 2024-02-10T09:33:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfba1d0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfba1d0700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:40860/0 (socket says 172.21.15.158:40860) 2024-02-10T09:33:18.204 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfba1d0700 1 -- 172.21.15.158:0/976906767 learned_addr learned my addr 172.21.15.158:0/976906767 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfba1d0700 1 -- 172.21.15.158:0/976906767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4005040 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfba1d0700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb41067b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fcfa4005010 tx=0x7fcfa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa4009320 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfa4013380 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb4106ea0 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa4014e50 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.201+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb4107330 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.202+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfa4014420 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.202+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb4045ad0 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.202+0000 7fcfaaffd700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcfa002ff10 0x7fcfa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.202+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfa4040eb0 con 0x7fcfb40f5410 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.203+0000 7fcfb99cf700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcfa002ff10 0x7fcfa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:18.205 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.203+0000 7fcfb99cf700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcfa002ff10 0x7fcfa00323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fcfb00096a0 tx=0x7fcfb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:18.208 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.207+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfa4026bf0 con 0x7fcfb40f5410 2024-02-10T09:33:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:16.871666+0000 mgr.smithi083.dowdym (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.414+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfb40fafa0 con 0x7fcfb40f5410 2024-02-10T09:33:18.416 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.415+0000 7fcfaaffd700 1 -- 172.21.15.158:0/976906767 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfa401db70 con 0x7fcfb40f5410 2024-02-10T09:33:18.416 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:18.416 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.416+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcfa002ff10 msgr2=0x7fcfa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.416+0000 7fcfbc434700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcfa002ff10 0x7fcfa00323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fcfb00096a0 tx=0x7fcfb0006b40).stop 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.416+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 msgr2=0x7fcfb41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.416+0000 7fcfbc434700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb41067b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fcfa4005010 tx=0x7fcfa400b040).stop 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.416+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 shutdown_connections 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.417+0000 7fcfbc434700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcfa002ff10 0x7fcfa00323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:18.418 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.417+0000 7fcfbc434700 1 --2- 172.21.15.158:0/976906767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcfb40f5410 0x7fcfb41067b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.417+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 >> 172.21.15.158:0/976906767 conn(0x7fcfb40f0b70 msgr2=0x7fcfb405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.417+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 shutdown_connections 2024-02-10T09:33:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:18.417+0000 7fcfbc434700 1 -- 172.21.15.158:0/976906767 wait complete. 2024-02-10T09:33:18.419 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:19.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:18 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:18.416314+0000 mon.smithi083 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.158:0/976906767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:20.148 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:20.148 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:20.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:19 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:18.871977+0000 mgr.smithi083.dowdym (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 -- 172.21.15.158:0/4051370241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 msgr2=0x7effcc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 --2- 172.21.15.158:0/4051370241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f4b60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7effc0009fb0 tx=0x7effc0009b30).stop 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 -- 172.21.15.158:0/4051370241 shutdown_connections 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 --2- 172.21.15.158:0/4051370241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f4b60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 -- 172.21.15.158:0/4051370241 >> 172.21.15.158:0/4051370241 conn(0x7effcc0f0260 msgr2=0x7effcc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 -- 172.21.15.158:0/4051370241 shutdown_connections 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.028+0000 7effd2cdf700 1 -- 172.21.15.158:0/4051370241 wait complete. 2024-02-10T09:33:22.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd2cdf700 1 Processor -- start 2024-02-10T09:33:22.031 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd2cdf700 1 -- start start 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd2cdf700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd2cdf700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0f5770 con 0x7effcc0f4740 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd0a7b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd0a7b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57920/0 (socket says 172.21.15.158:57920) 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.029+0000 7effd0a7b700 1 -- 172.21.15.158:0/557815601 learned_addr learned my addr 172.21.15.158:0/557815601 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effd0a7b700 1 -- 172.21.15.158:0/557815601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc0005040 con 0x7effcc0f4740 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effd0a7b700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f72f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7effc0005010 tx=0x7effc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc0009320 con 0x7effcc0f4740 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc0f5910 con 0x7effcc0f4740 2024-02-10T09:33:22.032 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc0013380 con 0x7effcc0f4740 2024-02-10T09:33:22.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc0014e50 con 0x7effcc0f4740 2024-02-10T09:33:22.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.030+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc0f5da0 con 0x7effcc0f4740 2024-02-10T09:33:22.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.031+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effc0014420 con 0x7effcc0f4740 2024-02-10T09:33:22.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.031+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effcc059100 con 0x7effcc0f4740 2024-02-10T09:33:22.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.031+0000 7effc97fa700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7effbc02ff10 0x7effbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:22.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.031+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effc0040e80 con 0x7effcc0f4740 2024-02-10T09:33:22.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.031+0000 7effcbfff700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7effbc02ff10 0x7effbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:22.034 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.032+0000 7effcbfff700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7effbc02ff10 0x7effbc0323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7effb80096a0 tx=0x7effb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:22.037 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.036+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effc0014790 con 0x7effcc0f4740 2024-02-10T09:33:22.243 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.242+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effcc0fc080 con 0x7effcc0f4740 2024-02-10T09:33:22.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.242+0000 7effc97fa700 1 -- 172.21.15.158:0/557815601 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effc001d380 con 0x7effcc0f4740 2024-02-10T09:33:22.244 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:22.244 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7effbc02ff10 msgr2=0x7effbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7effbc02ff10 0x7effbc0323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7effb80096a0 tx=0x7effb8006b40).stop 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 msgr2=0x7effcc0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f72f0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7effc0005010 tx=0x7effc000b040).stop 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 shutdown_connections 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7effbc02ff10 0x7effbc0323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 --2- 172.21.15.158:0/557815601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7effcc0f4740 0x7effcc0f72f0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 >> 172.21.15.158:0/557815601 conn(0x7effcc0f0260 msgr2=0x7effcc0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 shutdown_connections 2024-02-10T09:33:22.246 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:22.244+0000 7effd2cdf700 1 -- 172.21.15.158:0/557815601 wait complete. 2024-02-10T09:33:22.247 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:22.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:21 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:20.872357+0000 mgr.smithi083.dowdym (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:23.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:22.243857+0000 mon.smithi083 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.158:0/557815601' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:23.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:23.933 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:24.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:23 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:22.872674+0000 mgr.smithi083.dowdym (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 -- 172.21.15.158:0/3938182879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 msgr2=0x7fad480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 --2- 172.21.15.158:0/3938182879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad480f55c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fad30009fb0 tx=0x7fad30009b30).stop 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 -- 172.21.15.158:0/3938182879 shutdown_connections 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 --2- 172.21.15.158:0/3938182879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad480f55c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 -- 172.21.15.158:0/3938182879 >> 172.21.15.158:0/3938182879 conn(0x7fad480f09a0 msgr2=0x7fad480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 -- 172.21.15.158:0/3938182879 shutdown_connections 2024-02-10T09:33:25.791 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.789+0000 7fad4ce00700 1 -- 172.21.15.158:0/3938182879 wait complete. 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.790+0000 7fad4ce00700 1 Processor -- start 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.790+0000 7fad4ce00700 1 -- start start 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.790+0000 7fad4ce00700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad481092c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.790+0000 7fad4ce00700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad48060220 con 0x7fad480f51a0 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.790+0000 7fad477fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad481092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad477fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad481092c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57950/0 (socket says 172.21.15.158:57950) 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad477fe700 1 -- 172.21.15.158:0/2887200061 learned_addr learned my addr 172.21.15.158:0/2887200061 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad477fe700 1 -- 172.21.15.158:0/2887200061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad30005040 con 0x7fad480f51a0 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad477fe700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad481092c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fad30009830 tx=0x7fad3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad30009320 con 0x7fad480f51a0 2024-02-10T09:33:25.793 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad48060420 con 0x7fad480f51a0 2024-02-10T09:33:25.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad30013380 con 0x7fad480f51a0 2024-02-10T09:33:25.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad30014420 con 0x7fad480f51a0 2024-02-10T09:33:25.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.791+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad48109970 con 0x7fad480f51a0 2024-02-10T09:33:25.794 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.792+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad3001e420 con 0x7fad480f51a0 2024-02-10T09:33:25.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.792+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad48059100 con 0x7fad480f51a0 2024-02-10T09:33:25.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.792+0000 7fad3ffff700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fad3402ff60 0x7fad34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:25.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.792+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad30040e30 con 0x7fad480f51a0 2024-02-10T09:33:25.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.793+0000 7fad46ffd700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fad3402ff60 0x7fad34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:25.795 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.793+0000 7fad46ffd700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fad3402ff60 0x7fad34032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fad380096a0 tx=0x7fad38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:25.798 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:25.798+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad300095f0 con 0x7fad480f51a0 2024-02-10T09:33:26.002 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.001+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad48045ad0 con 0x7fad480f51a0 2024-02-10T09:33:26.003 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.002+0000 7fad3ffff700 1 -- 172.21.15.158:0/2887200061 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad30004490 con 0x7fad480f51a0 2024-02-10T09:33:26.003 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:26.003 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.003+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fad3402ff60 msgr2=0x7fad34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.003+0000 7fad4ce00700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fad3402ff60 0x7fad34032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fad380096a0 tx=0x7fad38006b40).stop 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.003+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 msgr2=0x7fad481092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.003+0000 7fad4ce00700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad481092c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fad30009830 tx=0x7fad3000b040).stop 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.004+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 shutdown_connections 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.004+0000 7fad4ce00700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fad3402ff60 0x7fad34032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.004+0000 7fad4ce00700 1 --2- 172.21.15.158:0/2887200061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad480f51a0 0x7fad481092c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:26.005 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.004+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 >> 172.21.15.158:0/2887200061 conn(0x7fad480f09a0 msgr2=0x7fad480f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:26.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.004+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 shutdown_connections 2024-02-10T09:33:26.006 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:26.004+0000 7fad4ce00700 1 -- 172.21.15.158:0/2887200061 wait complete. 2024-02-10T09:33:26.006 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:26.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:25 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:24.872946+0000 mgr.smithi083.dowdym (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:27.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:26 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:26.003277+0000 mon.smithi083 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.158:0/2887200061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:27.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:27.667 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:28.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:27 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:26.873341+0000 mgr.smithi083.dowdym (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:29.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 -- 172.21.15.158:0/2707918488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 msgr2=0x7f256c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:29.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 --2- 172.21.15.158:0/2707918488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c0f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f255c009fb0 tx=0x7f255c009b30).stop 2024-02-10T09:33:29.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 -- 172.21.15.158:0/2707918488 shutdown_connections 2024-02-10T09:33:29.627 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 --2- 172.21.15.158:0/2707918488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c0f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:29.628 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 -- 172.21.15.158:0/2707918488 >> 172.21.15.158:0/2707918488 conn(0x7f256c0f0b70 msgr2=0x7f256c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:29.628 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 -- 172.21.15.158:0/2707918488 shutdown_connections 2024-02-10T09:33:29.628 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 -- 172.21.15.158:0/2707918488 wait complete. 2024-02-10T09:33:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 Processor -- start 2024-02-10T09:33:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.626+0000 7f2572138700 1 -- start start 2024-02-10T09:33:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2572138700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c108a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2572138700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c108f10 con 0x7f256c0f5410 2024-02-10T09:33:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f256b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c108a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:29.629 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f256b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c108a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36388/0 (socket says 172.21.15.158:36388) 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f256b7fe700 1 -- 172.21.15.158:0/1198907149 learned_addr learned my addr 172.21.15.158:0/1198907149 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f256b7fe700 1 -- 172.21.15.158:0/1198907149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c005040 con 0x7f256c0f5410 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f256b7fe700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c108a20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f255c005010 tx=0x7f255c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f255c009320 con 0x7f256c0f5410 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256c109110 con 0x7f256c0f5410 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c1095a0 con 0x7f256c0f5410 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f255c013380 con 0x7f256c0f5410 2024-02-10T09:33:29.630 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.627+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f255c014df0 con 0x7f256c0f5410 2024-02-10T09:33:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.628+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f255c0095f0 con 0x7f256c0f5410 2024-02-10T09:33:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.628+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c045ad0 con 0x7f256c0f5410 2024-02-10T09:33:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.628+0000 7f2553fff700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f255402ff10 0x7f25540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.628+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f255c040ef0 con 0x7f256c0f5410 2024-02-10T09:33:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.629+0000 7f256affd700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f255402ff10 0x7f25540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:29.631 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.629+0000 7f256affd700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f255402ff10 0x7f25540323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f25600096a0 tx=0x7f2560006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:29.634 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.634+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f255c01dbb0 con 0x7f256c0f5410 2024-02-10T09:33:29.841 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.839+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f256c10a450 con 0x7f256c0f5410 2024-02-10T09:33:29.841 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.840+0000 7f2553fff700 1 -- 172.21.15.158:0/1198907149 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f255c01d9d0 con 0x7f256c0f5410 2024-02-10T09:33:29.841 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:29.842 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f255402ff10 msgr2=0x7f25540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f255402ff10 0x7f25540323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f25600096a0 tx=0x7f2560006b40).stop 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 msgr2=0x7f256c108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c108a20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f255c005010 tx=0x7f255c00b040).stop 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 shutdown_connections 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f255402ff10 0x7f25540323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 --2- 172.21.15.158:0/1198907149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f256c0f5410 0x7f256c108a20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 >> 172.21.15.158:0/1198907149 conn(0x7f256c0f0b70 msgr2=0x7f256c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 shutdown_connections 2024-02-10T09:33:29.843 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:29.842+0000 7f2572138700 1 -- 172.21.15.158:0/1198907149 wait complete. 2024-02-10T09:33:29.844 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:30.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:28.873727+0000 mgr.smithi083.dowdym (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:30.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:29 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:29.841392+0000 mon.smithi083 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.158:0/1198907149' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:31.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:31.502 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:32.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:31 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:30.874102+0000 mgr.smithi083.dowdym (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 -- 172.21.15.158:0/191865792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 msgr2=0x7fe0440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 --2- 172.21.15.158:0/191865792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe0440f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fe034009fb0 tx=0x7fe034009b30).stop 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 -- 172.21.15.158:0/191865792 shutdown_connections 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 --2- 172.21.15.158:0/191865792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe0440f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 -- 172.21.15.158:0/191865792 >> 172.21.15.158:0/191865792 conn(0x7fe0440f0b70 msgr2=0x7fe0440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 -- 172.21.15.158:0/191865792 shutdown_connections 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 -- 172.21.15.158:0/191865792 wait complete. 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 Processor -- start 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.435+0000 7fe049bb8700 1 -- start start 2024-02-10T09:33:33.437 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe049bb8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe044109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe049bb8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe044060240 con 0x7fe0440f5410 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe0437fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe044109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe0437fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe044109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36410/0 (socket says 172.21.15.158:36410) 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe0437fe700 1 -- 172.21.15.158:0/1915992224 learned_addr learned my addr 172.21.15.158:0/1915992224 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe0437fe700 1 -- 172.21.15.158:0/1915992224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034005040 con 0x7fe0440f5410 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe0437fe700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe044109520 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fe034005010 tx=0x7fe03400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe034009320 con 0x7fe0440f5410 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe044060440 con 0x7fe0440f5410 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe034013380 con 0x7fe0440f5410 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe034014e50 con 0x7fe0440f5410 2024-02-10T09:33:33.438 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.436+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe044109bd0 con 0x7fe0440f5410 2024-02-10T09:33:33.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.437+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe034014420 con 0x7fe0440f5410 2024-02-10T09:33:33.439 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.437+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe044045ad0 con 0x7fe0440f5410 2024-02-10T09:33:33.440 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.438+0000 7fe02bfff700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe02c02ff10 0x7fe02c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:33.440 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.438+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe034040e30 con 0x7fe0440f5410 2024-02-10T09:33:33.440 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.438+0000 7fe042ffd700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe02c02ff10 0x7fe02c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:33.440 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.438+0000 7fe042ffd700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe02c02ff10 0x7fe02c0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe0380096a0 tx=0x7fe038006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:33.442 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.442+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0340095f0 con 0x7fe0440f5410 2024-02-10T09:33:33.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.646+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe04410a3f0 con 0x7fe0440f5410 2024-02-10T09:33:33.647 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.647+0000 7fe02bfff700 1 -- 172.21.15.158:0/1915992224 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe03401db70 con 0x7fe0440f5410 2024-02-10T09:33:33.648 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:33.648 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:33.649 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe02c02ff10 msgr2=0x7fe02c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe02c02ff10 0x7fe02c0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe0380096a0 tx=0x7fe038006b40).stop 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 msgr2=0x7fe044109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe044109520 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fe034005010 tx=0x7fe03400b040).stop 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 shutdown_connections 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe02c02ff10 0x7fe02c0323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 --2- 172.21.15.158:0/1915992224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe0440f5410 0x7fe044109520 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 >> 172.21.15.158:0/1915992224 conn(0x7fe0440f0b70 msgr2=0x7fe0440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 shutdown_connections 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:33.649+0000 7fe049bb8700 1 -- 172.21.15.158:0/1915992224 wait complete. 2024-02-10T09:33:33.650 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:34.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:32.874384+0000 mgr.smithi083.dowdym (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:34.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:33 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:33.647658+0000 mon.smithi083 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.158:0/1915992224' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:35.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:35.359 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:36.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:35 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:34.874748+0000 mgr.smithi083.dowdym (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 -- 172.21.15.158:0/1957524398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 msgr2=0x7fd5940f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 --2- 172.21.15.158:0/1957524398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5940f9150 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fd584009fb0 tx=0x7fd584009b30).stop 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 -- 172.21.15.158:0/1957524398 shutdown_connections 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 --2- 172.21.15.158:0/1957524398 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5940f9150 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 -- 172.21.15.158:0/1957524398 >> 172.21.15.158:0/1957524398 conn(0x7fd5940f0b70 msgr2=0x7fd5940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 -- 172.21.15.158:0/1957524398 shutdown_connections 2024-02-10T09:33:37.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.259+0000 7fd599bdd700 1 -- 172.21.15.158:0/1957524398 wait complete. 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.260+0000 7fd599bdd700 1 Processor -- start 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.260+0000 7fd599bdd700 1 -- start start 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.260+0000 7fd599bdd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5941067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.260+0000 7fd599bdd700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594106c90 con 0x7fd594060220 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd5937fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5941067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd5937fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5941067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:36438/0 (socket says 172.21.15.158:36438) 2024-02-10T09:33:37.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd5937fe700 1 -- 172.21.15.158:0/308743550 learned_addr learned my addr 172.21.15.158:0/308743550 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:37.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd5937fe700 1 -- 172.21.15.158:0/308743550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584005040 con 0x7fd594060220 2024-02-10T09:33:37.263 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd5937fe700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5941067a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fd5840089f0 tx=0x7fd58400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd584009320 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd584013380 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd594106e90 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd584014e20 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.261+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd594107320 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.262+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5840263a0 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.262+0000 7fd58bfff700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd58002ff10 0x7fd5800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.262+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd584040f40 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.262+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd594059100 con 0x7fd594060220 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.263+0000 7fd592ffd700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd58002ff10 0x7fd5800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:37.264 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.263+0000 7fd592ffd700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd58002ff10 0x7fd5800323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd57c0096a0 tx=0x7fd57c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:37.267 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.267+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd584012920 con 0x7fd594060220 2024-02-10T09:33:37.471 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.471+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5940fc090 con 0x7fd594060220 2024-02-10T09:33:37.472 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.472+0000 7fd58bfff700 1 -- 172.21.15.158:0/308743550 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd58402b020 con 0x7fd594060220 2024-02-10T09:33:37.472 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:37.472 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd58002ff10 msgr2=0x7fd5800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd58002ff10 0x7fd5800323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd57c0096a0 tx=0x7fd57c006b40).stop 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 msgr2=0x7fd5941067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5941067a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fd5840089f0 tx=0x7fd58400b040).stop 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 shutdown_connections 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fd58002ff10 0x7fd5800323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 --2- 172.21.15.158:0/308743550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd594060220 0x7fd5941067a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 >> 172.21.15.158:0/308743550 conn(0x7fd5940f0b70 msgr2=0x7fd5940f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 shutdown_connections 2024-02-10T09:33:37.474 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:37.473+0000 7fd599bdd700 1 -- 172.21.15.158:0/308743550 wait complete. 2024-02-10T09:33:37.475 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:36.875069+0000 mgr.smithi083.dowdym (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:38.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:37 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:37.472076+0000 mon.smithi083 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.158:0/308743550' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:39.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:39.148 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:40.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:39 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:38.875377+0000 mgr.smithi083.dowdym (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:41.049 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.048+0000 7f4071d91700 1 -- 172.21.15.158:0/2141068022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 msgr2=0x7f406c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.048+0000 7f4071d91700 1 --2- 172.21.15.158:0/2141068022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c0f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f4060009fb0 tx=0x7f4060009b30).stop 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 -- 172.21.15.158:0/2141068022 shutdown_connections 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 --2- 172.21.15.158:0/2141068022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c0f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 -- 172.21.15.158:0/2141068022 >> 172.21.15.158:0/2141068022 conn(0x7f406c0f1c70 msgr2=0x7f406c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 -- 172.21.15.158:0/2141068022 shutdown_connections 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 -- 172.21.15.158:0/2141068022 wait complete. 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 Processor -- start 2024-02-10T09:33:41.050 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 -- start start 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.049+0000 7f4071d91700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c069e20 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f406b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f406b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57526/0 (socket says 172.21.15.158:57526) 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f406b7fe700 1 -- 172.21.15.158:0/2348449146 learned_addr learned my addr 172.21.15.158:0/2348449146 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f406b7fe700 1 -- 172.21.15.158:0/2348449146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060005040 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f406b7fe700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c06b9a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4060005010 tx=0x7f406000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4060009320 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c069fc0 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4060013380 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4060014e50 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.050+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c06a450 con 0x7f406c0f6510 2024-02-10T09:33:41.052 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.051+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4060014420 con 0x7f406c0f6510 2024-02-10T09:33:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.051+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c059100 con 0x7f406c0f6510 2024-02-10T09:33:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.051+0000 7f4053fff700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f405402ff10 0x7f40540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.051+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4060040ff0 con 0x7f406c0f6510 2024-02-10T09:33:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.051+0000 7f406affd700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f405402ff10 0x7f40540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:41.053 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.052+0000 7f406affd700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f405402ff10 0x7f40540323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f405c0096a0 tx=0x7f405c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:41.057 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.056+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4060014790 con 0x7f406c0f6510 2024-02-10T09:33:41.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.259+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f406c0fc0a0 con 0x7f406c0f6510 2024-02-10T09:33:41.259 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.259+0000 7f4053fff700 1 -- 172.21.15.158:0/2348449146 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f406001d980 con 0x7f406c0f6510 2024-02-10T09:33:41.260 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:41.260 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:41.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f405402ff10 msgr2=0x7f40540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:41.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f405402ff10 0x7f40540323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f405c0096a0 tx=0x7f405c006b40).stop 2024-02-10T09:33:41.261 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 msgr2=0x7f406c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c06b9a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f4060005010 tx=0x7f406000b040).stop 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 shutdown_connections 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f405402ff10 0x7f40540323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 --2- 172.21.15.158:0/2348449146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f406c0f6510 0x7f406c06b9a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 >> 172.21.15.158:0/2348449146 conn(0x7f406c0f1c70 msgr2=0x7f406c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 shutdown_connections 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:41.261+0000 7f4071d91700 1 -- 172.21.15.158:0/2348449146 wait complete. 2024-02-10T09:33:41.262 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:42.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:40.875739+0000 mgr.smithi083.dowdym (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:42.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:41 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:41.259855+0000 mon.smithi083 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.158:0/2348449146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:42.931 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:42.932 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:44.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:43 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:42.876140+0000 mgr.smithi083.dowdym (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 -- 172.21.15.158:0/87715238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 msgr2=0x7fe9440f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 --2- 172.21.15.158:0/87715238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe9440f55c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fe940009fb0 tx=0x7fe940009b30).stop 2024-02-10T09:33:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 -- 172.21.15.158:0/87715238 shutdown_connections 2024-02-10T09:33:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 --2- 172.21.15.158:0/87715238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe9440f55c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 -- 172.21.15.158:0/87715238 >> 172.21.15.158:0/87715238 conn(0x7fe9440f09a0 msgr2=0x7fe9440f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:45.027 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 -- 172.21.15.158:0/87715238 shutdown_connections 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 -- 172.21.15.158:0/87715238 wait complete. 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 Processor -- start 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.026+0000 7fe94b3b6700 1 -- start start 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94b3b6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe944109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94b3b6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe944060280 con 0x7fe9440f51a0 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94a3b4700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe944109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94a3b4700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe944109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57550/0 (socket says 172.21.15.158:57550) 2024-02-10T09:33:45.028 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94a3b4700 1 -- 172.21.15.158:0/1820067198 learned_addr learned my addr 172.21.15.158:0/1820067198 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:45.029 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94a3b4700 1 -- 172.21.15.158:0/1820067198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe940005040 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94a3b4700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe944109290 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fe940009830 tx=0x7fe94000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe940009320 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe944060480 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe940013380 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.027+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe940014420 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.028+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9441099c0 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.028+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe944045ad0 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.028+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe94001e420 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.029+0000 7fe93affd700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe93002ff60 0x7fe930032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.029+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe940040e20 con 0x7fe9440f51a0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.029+0000 7fe949bb3700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe93002ff60 0x7fe930032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:45.030 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.029+0000 7fe949bb3700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe93002ff60 0x7fe930032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe9340096a0 tx=0x7fe934006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:45.033 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.033+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe94001c930 con 0x7fe9440f51a0 2024-02-10T09:33:45.241 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.240+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe94410a220 con 0x7fe9440f51a0 2024-02-10T09:33:45.241 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.241+0000 7fe93affd700 1 -- 172.21.15.158:0/1820067198 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe94002c2d0 con 0x7fe9440f51a0 2024-02-10T09:33:45.242 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:45.242 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe93002ff60 msgr2=0x7fe930032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe93002ff60 0x7fe930032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fe9340096a0 tx=0x7fe934006b40).stop 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 msgr2=0x7fe944109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe944109290 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fe940009830 tx=0x7fe94000b040).stop 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 shutdown_connections 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe93002ff60 0x7fe930032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 --2- 172.21.15.158:0/1820067198 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9440f51a0 0x7fe944109290 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 >> 172.21.15.158:0/1820067198 conn(0x7fe9440f09a0 msgr2=0x7fe9440f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 shutdown_connections 2024-02-10T09:33:45.244 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:45.243+0000 7fe94b3b6700 1 -- 172.21.15.158:0/1820067198 wait complete. 2024-02-10T09:33:45.245 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:46.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:44.876448+0000 mgr.smithi083.dowdym (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:46.026 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:45 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:45.241610+0000 mon.smithi083 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.158:0/1820067198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:46.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:46.963 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:48.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:47 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:46.876802+0000 mgr.smithi083.dowdym (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.782+0000 7f4415d80700 1 -- 172.21.15.158:0/1780137448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 msgr2=0x7f44100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.782+0000 7f4415d80700 1 --2- 172.21.15.158:0/1780137448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f44100f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f4400009fb0 tx=0x7f4400009b30).stop 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.782+0000 7f4415d80700 1 -- 172.21.15.158:0/1780137448 shutdown_connections 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.782+0000 7f4415d80700 1 --2- 172.21.15.158:0/1780137448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f44100f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.782+0000 7f4415d80700 1 -- 172.21.15.158:0/1780137448 >> 172.21.15.158:0/1780137448 conn(0x7f44100f0b70 msgr2=0x7f44100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.783+0000 7f4415d80700 1 -- 172.21.15.158:0/1780137448 shutdown_connections 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.783+0000 7f4415d80700 1 -- 172.21.15.158:0/1780137448 wait complete. 2024-02-10T09:33:48.785 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.783+0000 7f4415d80700 1 Processor -- start 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.783+0000 7f4415d80700 1 -- start start 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.783+0000 7f4415d80700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f4410109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.783+0000 7f4415d80700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4410060220 con 0x7f44100f5410 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f440f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f4410109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f440f7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f4410109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:57558/0 (socket says 172.21.15.158:57558) 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f440f7fe700 1 -- 172.21.15.158:0/936890348 learned_addr learned my addr 172.21.15.158:0/936890348 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f440f7fe700 1 -- 172.21.15.158:0/936890348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400005040 con 0x7f44100f5410 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f440f7fe700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f4410109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f44000089f0 tx=0x7f440000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4400009320 con 0x7f44100f5410 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4400013380 con 0x7f44100f5410 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4410060420 con 0x7f44100f5410 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f440001ed80 con 0x7f44100f5410 2024-02-10T09:33:48.786 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.784+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4410109bb0 con 0x7f44100f5410 2024-02-10T09:33:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.785+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f440001e420 con 0x7f44100f5410 2024-02-10T09:33:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.785+0000 7f4407fff700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f43fc02ff60 0x7f43fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.785+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4400041050 con 0x7f44100f5410 2024-02-10T09:33:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.785+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4410059100 con 0x7f44100f5410 2024-02-10T09:33:48.787 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.786+0000 7f440effd700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f43fc02ff60 0x7f43fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:48.788 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.786+0000 7f440effd700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f43fc02ff60 0x7f43fc032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f43f80096a0 tx=0x7f43f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:48.790 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.790+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f440001cb10 con 0x7f44100f5410 2024-02-10T09:33:48.996 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.996+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44100fafa0 con 0x7f44100f5410 2024-02-10T09:33:48.997 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.996+0000 7f4407fff700 1 -- 172.21.15.158:0/936890348 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f440001cb10 con 0x7f44100f5410 2024-02-10T09:33:48.997 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:48.997 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f43fc02ff60 msgr2=0x7f43fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f43fc02ff60 0x7f43fc032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f43f80096a0 tx=0x7f43f8006b40).stop 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 msgr2=0x7f4410109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f4410109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f44000089f0 tx=0x7f440000b040).stop 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 shutdown_connections 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f43fc02ff60 0x7f43fc032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 --2- 172.21.15.158:0/936890348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f44100f5410 0x7f4410109500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 >> 172.21.15.158:0/936890348 conn(0x7f44100f0b70 msgr2=0x7f44100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:48.999 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 shutdown_connections 2024-02-10T09:33:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:48.998+0000 7f4415d80700 1 -- 172.21.15.158:0/936890348 wait complete. 2024-02-10T09:33:49.000 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:50.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:48.877105+0000 mgr.smithi083.dowdym (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:50.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:49 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:48.996979+0000 mon.smithi083 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.158:0/936890348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:50.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:50.644 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:52.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:51 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:50.877492+0000 mgr.smithi083.dowdym (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 -- 172.21.15.158:0/2240908080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 msgr2=0x7f04d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 --2- 172.21.15.158:0/2240908080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f04c8009fb0 tx=0x7f04c8009b30).stop 2024-02-10T09:33:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 -- 172.21.15.158:0/2240908080 shutdown_connections 2024-02-10T09:33:52.726 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 --2- 172.21.15.158:0/2240908080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 -- 172.21.15.158:0/2240908080 >> 172.21.15.158:0/2240908080 conn(0x7f04d40f0b70 msgr2=0x7f04d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 -- 172.21.15.158:0/2240908080 shutdown_connections 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.724+0000 7f04d94a2700 1 -- 172.21.15.158:0/2240908080 wait complete. 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d94a2700 1 Processor -- start 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d94a2700 1 -- start start 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d94a2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d94a2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d4109530 con 0x7f04d40f5410 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d2ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:41642/0 (socket says 172.21.15.158:41642) 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.725+0000 7f04d2ffd700 1 -- 172.21.15.158:0/3239728289 learned_addr learned my addr 172.21.15.158:0/3239728289 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04d2ffd700 1 -- 172.21.15.158:0/3239728289 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c8005040 con 0x7f04d40f5410 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04d2ffd700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40601b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f04c8005010 tx=0x7f04c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04c8009320 con 0x7f04d40f5410 2024-02-10T09:33:52.727 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04c8013380 con 0x7f04d40f5410 2024-02-10T09:33:52.728 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d4109670 con 0x7f04d40f5410 2024-02-10T09:33:52.728 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04c8014e50 con 0x7f04d40f5410 2024-02-10T09:33:52.728 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.726+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d4109b60 con 0x7f04d40f5410 2024-02-10T09:33:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.727+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04c80263a0 con 0x7f04d40f5410 2024-02-10T09:33:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.727+0000 7f04bb7fe700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f04bc02ff10 0x7f04bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.727+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04c8040e50 con 0x7f04d40f5410 2024-02-10T09:33:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.727+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d4045ad0 con 0x7f04d40f5410 2024-02-10T09:33:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.727+0000 7f04d27fc700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f04bc02ff10 0x7f04bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:52.729 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.728+0000 7f04d27fc700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f04bc02ff10 0x7f04bc0323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f04c40096a0 tx=0x7f04c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:52.732 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.732+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04c8012920 con 0x7f04d40f5410 2024-02-10T09:33:52.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.939+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04d410a290 con 0x7f04d40f5410 2024-02-10T09:33:52.940 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.940+0000 7f04bb7fe700 1 -- 172.21.15.158:0/3239728289 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f04c802b020 con 0x7f04d40f5410 2024-02-10T09:33:52.940 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:52.940 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:52.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.941+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f04bc02ff10 msgr2=0x7f04bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:52.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.941+0000 7f04d94a2700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f04bc02ff10 0x7f04bc0323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f04c40096a0 tx=0x7f04c4006b40).stop 2024-02-10T09:33:52.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 msgr2=0x7f04d40601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:52.942 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40601b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f04c8005010 tx=0x7f04c800b040).stop 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 shutdown_connections 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f04bc02ff10 0x7f04bc0323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 --2- 172.21.15.158:0/3239728289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04d40f5410 0x7f04d40601b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 >> 172.21.15.158:0/3239728289 conn(0x7f04d40f0b70 msgr2=0x7f04d40f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 shutdown_connections 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:52.942+0000 7f04d94a2700 1 -- 172.21.15.158:0/3239728289 wait complete. 2024-02-10T09:33:52.943 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:54.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:52.877925+0000 mgr.smithi083.dowdym (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:54.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:53 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:52.940324+0000 mon.smithi083 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.158:0/3239728289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:54.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:54.713 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:33:56.025 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:55 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:54.878189+0000 mgr.smithi083.dowdym (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 -- 172.21.15.158:0/3259599535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 msgr2=0x7f3fdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 --2- 172.21.15.158:0/3259599535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc0f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f3fd0009fb0 tx=0x7f3fd0009b30).stop 2024-02-10T09:33:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 -- 172.21.15.158:0/3259599535 shutdown_connections 2024-02-10T09:33:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 --2- 172.21.15.158:0/3259599535 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc0f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 -- 172.21.15.158:0/3259599535 >> 172.21.15.158:0/3259599535 conn(0x7f3fdc0f0b70 msgr2=0x7f3fdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:56.708 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 -- 172.21.15.158:0/3259599535 shutdown_connections 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 -- 172.21.15.158:0/3259599535 wait complete. 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 Processor -- start 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.706+0000 7f3fe34f9700 1 -- start start 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe34f9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe34f9700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc105bd0 con 0x7f3fdc0f5410 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe1295700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe1295700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:41662/0 (socket says 172.21.15.158:41662) 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe1295700 1 -- 172.21.15.158:0/1012542375 learned_addr learned my addr 172.21.15.158:0/1012542375 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe1295700 1 -- 172.21.15.158:0/1012542375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0005040 con 0x7f3fdc0f5410 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fe1295700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc1056e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f3fd0005010 tx=0x7f3fd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.707+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd0009320 con 0x7f3fdc0f5410 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.708+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fd0013380 con 0x7f3fdc0f5410 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.708+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc105dd0 con 0x7f3fdc0f5410 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.708+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd0014e50 con 0x7f3fdc0f5410 2024-02-10T09:33:56.709 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.708+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc106260 con 0x7f3fdc0f5410 2024-02-10T09:33:56.710 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.709+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fd00095f0 con 0x7f3fdc0f5410 2024-02-10T09:33:56.710 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.709+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc045ad0 con 0x7f3fdc0f5410 2024-02-10T09:33:56.711 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.709+0000 7f3fcdffb700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3fc802ff10 0x7f3fc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:33:56.711 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.709+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fd0040e70 con 0x7f3fdc0f5410 2024-02-10T09:33:56.711 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.709+0000 7f3fe0a94700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3fc802ff10 0x7f3fc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:33:56.711 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.709+0000 7f3fe0a94700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3fc802ff10 0x7f3fc80323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3fd80096a0 tx=0x7f3fd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:33:56.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.714+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fd0014420 con 0x7f3fdc0f5410 2024-02-10T09:33:56.921 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.921+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fdc1070b0 con 0x7f3fdc0f5410 2024-02-10T09:33:56.921 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.921+0000 7f3fcdffb700 1 -- 172.21.15.158:0/1012542375 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fd0010230 con 0x7f3fdc0f5410 2024-02-10T09:33:56.922 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:33:56.922 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:33:56.923 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3fc802ff10 msgr2=0x7f3fc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:56.923 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3fc802ff10 0x7f3fc80323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f3fd80096a0 tx=0x7f3fd8006b40).stop 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 msgr2=0x7f3fdc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc1056e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f3fd0005010 tx=0x7f3fd000b040).stop 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 shutdown_connections 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f3fc802ff10 0x7f3fc80323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 --2- 172.21.15.158:0/1012542375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3fdc0f5410 0x7f3fdc1056e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 >> 172.21.15.158:0/1012542375 conn(0x7f3fdc0f0b70 msgr2=0x7f3fdc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 shutdown_connections 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:33:56.923+0000 7f3fe34f9700 1 -- 172.21.15.158:0/1012542375 wait complete. 2024-02-10T09:33:56.924 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:33:57.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:56 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:33:56.921771+0000 mon.smithi083 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.158:0/1012542375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:33:58.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:57 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:56.878609+0000 mgr.smithi083.dowdym (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:33:58.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:33:58.482 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:00.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:33:59 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:33:58.878970+0000 mgr.smithi083.dowdym (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.475+0000 7f5edc705700 1 -- 172.21.15.158:0/3518977131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 msgr2=0x7f5ed40f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.475+0000 7f5edc705700 1 --2- 172.21.15.158:0/3518977131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f9190 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f5ec8009fb0 tx=0x7f5ec8009b30).stop 2024-02-10T09:34:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 -- 172.21.15.158:0/3518977131 shutdown_connections 2024-02-10T09:34:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 --2- 172.21.15.158:0/3518977131 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f9190 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 -- 172.21.15.158:0/3518977131 >> 172.21.15.158:0/3518977131 conn(0x7f5ed40f0b70 msgr2=0x7f5ed40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:00.479 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 -- 172.21.15.158:0/3518977131 shutdown_connections 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 -- 172.21.15.158:0/3518977131 wait complete. 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 Processor -- start 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 -- start start 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.476+0000 7f5edc705700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40f5090 con 0x7f5ed40601e0 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5eda4a1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5eda4a1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33850/0 (socket says 172.21.15.158:33850) 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5eda4a1700 1 -- 172.21.15.158:0/3959503217 learned_addr learned my addr 172.21.15.158:0/3959503217 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5eda4a1700 1 -- 172.21.15.158:0/3959503217 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec8005040 con 0x7f5ed40601e0 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5eda4a1700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f6c10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f5ec8007a10 tx=0x7f5ec8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:00.480 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ec8018070 con 0x7f5ed40601e0 2024-02-10T09:34:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed40f5230 con 0x7f5ed40601e0 2024-02-10T09:34:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ec8013410 con 0x7f5ed40601e0 2024-02-10T09:34:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ec801c3c0 con 0x7f5ed40601e0 2024-02-10T09:34:00.481 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.477+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed40f56c0 con 0x7f5ed40601e0 2024-02-10T09:34:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.478+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ec8013ca0 con 0x7f5ed40601e0 2024-02-10T09:34:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.478+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed4059100 con 0x7f5ed40601e0 2024-02-10T09:34:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.478+0000 7f5ec6ffd700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5ec002ff60 0x7f5ec0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.478+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ec80414b0 con 0x7f5ed40601e0 2024-02-10T09:34:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.479+0000 7f5ed9ca0700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5ec002ff60 0x7f5ec0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:00.482 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.479+0000 7f5ed9ca0700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5ec002ff60 0x7f5ec0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5ed00096a0 tx=0x7f5ed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:00.484 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.483+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ec801d420 con 0x7f5ed40601e0 2024-02-10T09:34:00.689 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.688+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ed4045ad0 con 0x7f5ed40601e0 2024-02-10T09:34:00.690 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.689+0000 7f5ec6ffd700 1 -- 172.21.15.158:0/3959503217 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5ec8025ce0 con 0x7f5ed40601e0 2024-02-10T09:34:00.690 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:00.690 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5ec002ff60 msgr2=0x7f5ec0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5ec002ff60 0x7f5ec0032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f5ed00096a0 tx=0x7f5ed0006b40).stop 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 msgr2=0x7f5ed40f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f6c10 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f5ec8007a10 tx=0x7f5ec8009b30).stop 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 shutdown_connections 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f5ec002ff60 0x7f5ec0032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:00.692 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 --2- 172.21.15.158:0/3959503217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ed40601e0 0x7f5ed40f6c10 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:00.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 >> 172.21.15.158:0/3959503217 conn(0x7f5ed40f0b70 msgr2=0x7f5ed40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:00.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 shutdown_connections 2024-02-10T09:34:00.693 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:00.690+0000 7f5edc705700 1 -- 172.21.15.158:0/3959503217 wait complete. 2024-02-10T09:34:00.693 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:01.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:00 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:00.690154+0000 mon.smithi083 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.158:0/3959503217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:02.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:01 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:00.879379+0000 mgr.smithi083.dowdym (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:02.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:34:02.492 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:04.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:03 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:02.879733+0000 mgr.smithi083.dowdym (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 -- 172.21.15.158:0/14050057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 msgr2=0x7f87a80f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 --2- 172.21.15.158:0/14050057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a80f6ea0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f8798009fb0 tx=0x7f8798009b30).stop 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 -- 172.21.15.158:0/14050057 shutdown_connections 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 --2- 172.21.15.158:0/14050057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a80f6ea0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 -- 172.21.15.158:0/14050057 >> 172.21.15.158:0/14050057 conn(0x7f87a80f02e0 msgr2=0x7f87a80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 -- 172.21.15.158:0/14050057 shutdown_connections 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.276+0000 7f87ae944700 1 -- 172.21.15.158:0/14050057 wait complete. 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87ae944700 1 Processor -- start 2024-02-10T09:34:04.278 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87ae944700 1 -- start start 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87ae944700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a810b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87ae944700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a81072b0 con 0x7f87a80f6a80 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87a7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a810b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87a7fff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a810b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33866/0 (socket says 172.21.15.158:33866) 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.277+0000 7f87a7fff700 1 -- 172.21.15.158:0/98560236 learned_addr learned my addr 172.21.15.158:0/98560236 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87a7fff700 1 -- 172.21.15.158:0/98560236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798005040 con 0x7f87a80f6a80 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87a7fff700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a810b8c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f87980089f0 tx=0x7f879800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:04.280 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8798009320 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8798013380 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a80f5310 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8798014e20 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.278+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a810bdb0 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.279+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8798014420 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.279+0000 7f87a4ff9700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f879002ff10 0x7f87900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.279+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a8045ad0 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.279+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8798040e70 con 0x7f87a80f6a80 2024-02-10T09:34:04.281 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.279+0000 7f87a77fe700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f879002ff10 0x7f87900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:04.282 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.280+0000 7f87a77fe700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f879002ff10 0x7f87900323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f879c0096a0 tx=0x7f879c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:04.285 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.284+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f879801d980 con 0x7f87a80f6a80 2024-02-10T09:34:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.487+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87a810c570 con 0x7f87a80f6a80 2024-02-10T09:34:04.489 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.488+0000 7f87a4ff9700 1 -- 172.21.15.158:0/98560236 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f879801d380 con 0x7f87a80f6a80 2024-02-10T09:34:04.490 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:04.490 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:04.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f879002ff10 msgr2=0x7f87900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:04.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f879002ff10 0x7f87900323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f879c0096a0 tx=0x7f879c006b40).stop 2024-02-10T09:34:04.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 msgr2=0x7f87a810b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:04.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a810b8c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f87980089f0 tx=0x7f879800b040).stop 2024-02-10T09:34:04.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 shutdown_connections 2024-02-10T09:34:04.491 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f879002ff10 0x7f87900323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:04.492 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 --2- 172.21.15.158:0/98560236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f87a80f6a80 0x7f87a810b8c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:04.492 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 >> 172.21.15.158:0/98560236 conn(0x7f87a80f02e0 msgr2=0x7f87a80f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:04.492 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 shutdown_connections 2024-02-10T09:34:04.492 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:04.490+0000 7f87ae944700 1 -- 172.21.15.158:0/98560236 wait complete. 2024-02-10T09:34:04.492 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:05.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:04 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:04.489501+0000 mon.smithi083 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.158:0/98560236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:06.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:34:06.181 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:06.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:05 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:04.880112+0000 mgr.smithi083.dowdym (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 -- 172.21.15.158:0/3103689080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 msgr2=0x7fe80c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 --2- 172.21.15.158:0/3103689080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c0f90f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fe800009fb0 tx=0x7fe800009b30).stop 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 -- 172.21.15.158:0/3103689080 shutdown_connections 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 --2- 172.21.15.158:0/3103689080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c0f90f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 -- 172.21.15.158:0/3103689080 >> 172.21.15.158:0/3103689080 conn(0x7fe80c0f0b70 msgr2=0x7fe80c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 -- 172.21.15.158:0/3103689080 shutdown_connections 2024-02-10T09:34:08.015 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.013+0000 7fe8120de700 1 -- 172.21.15.158:0/3103689080 wait complete. 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.014+0000 7fe8120de700 1 Processor -- start 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.014+0000 7fe8120de700 1 -- start start 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe8120de700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe8120de700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c106c70 con 0x7fe80c060280 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe80b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe80b7fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:33886/0 (socket says 172.21.15.158:33886) 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe80b7fe700 1 -- 172.21.15.158:0/2666667199 learned_addr learned my addr 172.21.15.158:0/2666667199 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:08.017 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe80b7fe700 1 -- 172.21.15.158:0/2666667199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe800005040 con 0x7fe80c060280 2024-02-10T09:34:08.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe80b7fe700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c106780 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe800009830 tx=0x7fe80000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:08.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe800009320 con 0x7fe80c060280 2024-02-10T09:34:08.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c106e70 con 0x7fe80c060280 2024-02-10T09:34:08.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe800013380 con 0x7fe80c060280 2024-02-10T09:34:08.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.015+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe800014420 con 0x7fe80c060280 2024-02-10T09:34:08.018 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.016+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c107300 con 0x7fe80c060280 2024-02-10T09:34:08.019 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.016+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe80001e420 con 0x7fe80c060280 2024-02-10T09:34:08.019 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.016+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80c045ad0 con 0x7fe80c060280 2024-02-10T09:34:08.019 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.017+0000 7fe7f3fff700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7f402ffb0 0x7fe7f4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:08.019 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.017+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe800040eb0 con 0x7fe80c060280 2024-02-10T09:34:08.019 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.017+0000 7fe80affd700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7f402ffb0 0x7fe7f4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:08.019 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.017+0000 7fe80affd700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7f402ffb0 0x7fe7f4032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe7fc0096a0 tx=0x7fe7fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:08.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.021+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe80001cb10 con 0x7fe80c060280 2024-02-10T09:34:08.226 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.224+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe80c0fc090 con 0x7fe80c060280 2024-02-10T09:34:08.227 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.225+0000 7fe7f3fff700 1 -- 172.21.15.158:0/2666667199 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe80002c2d0 con 0x7fe80c060280 2024-02-10T09:34:08.227 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:08.227 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:08.228 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7f402ffb0 msgr2=0x7fe7f4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7f402ffb0 0x7fe7f4032470 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fe7fc0096a0 tx=0x7fe7fc006b40).stop 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 msgr2=0x7fe80c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c106780 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fe800009830 tx=0x7fe80000b040).stop 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 shutdown_connections 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fe7f402ffb0 0x7fe7f4032470 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 --2- 172.21.15.158:0/2666667199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe80c060280 0x7fe80c106780 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 >> 172.21.15.158:0/2666667199 conn(0x7fe80c0f0b70 msgr2=0x7fe80c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 shutdown_connections 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:08.227+0000 7fe8120de700 1 -- 172.21.15.158:0/2666667199 wait complete. 2024-02-10T09:34:08.229 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:08.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:07 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:06.880393+0000 mgr.smithi083.dowdym (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:09.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:08 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:08.226711+0000 mon.smithi083 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.158:0/2666667199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:09.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:34:09.723 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:10.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:09 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:08.880730+0000 mgr.smithi083.dowdym (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- 172.21.15.158:0/848460631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 msgr2=0x7fcc0c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 --2- 172.21.15.158:0/848460631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c0f55c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fcbfc009fb0 tx=0x7fcbfc009b30).stop 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- 172.21.15.158:0/848460631 shutdown_connections 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 --2- 172.21.15.158:0/848460631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c0f55c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- 172.21.15.158:0/848460631 >> 172.21.15.158:0/848460631 conn(0x7fcc0c0f09a0 msgr2=0x7fcc0c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- 172.21.15.158:0/848460631 shutdown_connections 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- 172.21.15.158:0/848460631 wait complete. 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 Processor -- start 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- start start 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.710+0000 7fcc12fbd700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1059c0 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.713 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc11fbb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc11fbb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39664/0 (socket says 172.21.15.158:39664) 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc11fbb700 1 -- 172.21.15.158:0/3508316145 learned_addr learned my addr 172.21.15.158:0/3508316145 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc11fbb700 1 -- 172.21.15.158:0/3508316145 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc005040 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc11fbb700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c1054d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fcbfc0089f0 tx=0x7fcbfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbfc009320 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c105bc0 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbfc013380 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbfc014e20 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.714 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.711+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c106050 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.712+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf4004fa0 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.712+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbfc0095f0 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.712+0000 7fcc027fc700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcbf802ff10 0x7fcbf80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.712+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbfc040e30 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.712+0000 7fcc117ba700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcbf802ff10 0x7fcbf80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:11.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.713+0000 7fcc117ba700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcbf802ff10 0x7fcbf80323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fcc080096a0 tx=0x7fcc08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:11.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.717+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbfc014420 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.922 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.921+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbf4005670 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.923 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.921+0000 7fcc027fc700 1 -- 172.21.15.158:0/3508316145 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcbfc010230 con 0x7fcc0c0f51a0 2024-02-10T09:34:11.923 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:11.923 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcbf802ff10 msgr2=0x7fcbf80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcbf802ff10 0x7fcbf80323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fcc080096a0 tx=0x7fcc08006b40).stop 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 msgr2=0x7fcc0c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c1054d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fcbfc0089f0 tx=0x7fcbfc00b040).stop 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 shutdown_connections 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7fcbf802ff10 0x7fcbf80323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 --2- 172.21.15.158:0/3508316145 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc0c0f51a0 0x7fcc0c1054d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:11.925 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.923+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 >> 172.21.15.158:0/3508316145 conn(0x7fcc0c0f09a0 msgr2=0x7fcc0c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:11.926 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.924+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 shutdown_connections 2024-02-10T09:34:11.926 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:11.924+0000 7fcc12fbd700 1 -- 172.21.15.158:0/3508316145 wait complete. 2024-02-10T09:34:11.926 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:10.881002+0000 mgr.smithi083.dowdym (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:12.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:11 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:11.922858+0000 mon.smithi083 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.158:0/3508316145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:13.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:34:13.691 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:14.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:13 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:12.881366+0000 mgr.smithi083.dowdym (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:15.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:14.178917+0000 mon.smithi083 (mon.0) 338 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:34:15.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:14 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:14.185693+0000 mon.smithi083 (mon.0) 339 : audit [INF] from='mgr.14162 172.21.15.83:0/2243643483' entity='mgr.smithi083.dowdym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi083.dowdym/trash_purge_schedule"}]: dispatch 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 -- 172.21.15.158:0/631003232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 msgr2=0x7f40e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 --2- 172.21.15.158:0/631003232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e00f66c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f40d0009fb0 tx=0x7f40d0009b30).stop 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 -- 172.21.15.158:0/631003232 shutdown_connections 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 --2- 172.21.15.158:0/631003232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e00f66c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 -- 172.21.15.158:0/631003232 >> 172.21.15.158:0/631003232 conn(0x7f40e00f1aa0 msgr2=0x7f40e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 -- 172.21.15.158:0/631003232 shutdown_connections 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 -- 172.21.15.158:0/631003232 wait complete. 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 Processor -- start 2024-02-10T09:34:15.820 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.802+0000 7f40e5344700 1 -- start start 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40e5344700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40e5344700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e0106ac0 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:39684/0 (socket says 172.21.15.158:39684) 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dffff700 1 -- 172.21.15.158:0/1966847972 learned_addr learned my addr 172.21.15.158:0/1966847972 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dffff700 1 -- 172.21.15.158:0/1966847972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0005040 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dffff700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e01065d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f40d00089f0 tx=0x7f40d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d0009320 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40d0013380 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e0106cc0 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d0014e20 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.803+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e0107150 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.804+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40d00263a0 con 0x7f40e00f62a0 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.804+0000 7f40dcff9700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f40c802ff10 0x7f40c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:15.821 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.804+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40d0040f40 con 0x7f40e00f62a0 2024-02-10T09:34:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.804+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40e0045ad0 con 0x7f40e00f62a0 2024-02-10T09:34:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.805+0000 7f40df7fe700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f40c802ff10 0x7f40c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.805+0000 7f40df7fe700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f40c802ff10 0x7f40c80323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f40d40096a0 tx=0x7f40d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:15.822 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:15.809+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40d0012920 con 0x7f40e00f62a0 2024-02-10T09:34:16.020 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.018+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40e0107e70 con 0x7f40e00f62a0 2024-02-10T09:34:16.021 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.020+0000 7f40dcff9700 1 -- 172.21.15.158:0/1966847972 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40d002b020 con 0x7f40e00f62a0 2024-02-10T09:34:16.021 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:16.021 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:16.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f40c802ff10 msgr2=0x7f40c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:16.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f40c802ff10 0x7f40c80323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f40d40096a0 tx=0x7f40d4006b40).stop 2024-02-10T09:34:16.022 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 msgr2=0x7f40e01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e01065d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f40d00089f0 tx=0x7f40d000b040).stop 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 shutdown_connections 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f40c802ff10 0x7f40c80323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 --2- 172.21.15.158:0/1966847972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40e00f62a0 0x7f40e01065d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 >> 172.21.15.158:0/1966847972 conn(0x7f40e00f1aa0 msgr2=0x7f40e00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 shutdown_connections 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:16.021+0000 7f40e5344700 1 -- 172.21.15.158:0/1966847972 wait complete. 2024-02-10T09:34:16.023 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:16.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:15 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:14.881714+0000 mgr.smithi083.dowdym (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:17.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:16 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:16.020559+0000 mon.smithi083 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.158:0/1966847972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:17.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:34:17.650 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:18.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:17 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:16.882076+0000 mgr.smithi083.dowdym (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:19.506 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 -- 172.21.15.158:0/3489516738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 msgr2=0x7f353c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:19.506 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 --2- 172.21.15.158:0/3489516738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c0f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f352c009fb0 tx=0x7f352c009b30).stop 2024-02-10T09:34:19.506 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 -- 172.21.15.158:0/3489516738 shutdown_connections 2024-02-10T09:34:19.506 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 --2- 172.21.15.158:0/3489516738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c0f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:19.506 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 -- 172.21.15.158:0/3489516738 >> 172.21.15.158:0/3489516738 conn(0x7f353c0f0b70 msgr2=0x7f353c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 -- 172.21.15.158:0/3489516738 shutdown_connections 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.502+0000 7f35433b2700 1 -- 172.21.15.158:0/3489516738 wait complete. 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f35433b2700 1 Processor -- start 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f35433b2700 1 -- start start 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f35433b2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f35433b2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c060240 con 0x7f353c0f5410 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f354114e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f354114e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56960/0 (socket says 172.21.15.158:56960) 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f354114e700 1 -- 172.21.15.158:0/3604365844 learned_addr learned my addr 172.21.15.158:0/3604365844 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f354114e700 1 -- 172.21.15.158:0/3604365844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c005040 con 0x7f353c0f5410 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.503+0000 7f354114e700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c1094e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f352c005010 tx=0x7f352c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.504+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f352c009320 con 0x7f353c0f5410 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.504+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c060440 con 0x7f353c0f5410 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.504+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f352c013380 con 0x7f353c0f5410 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.504+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f352c014e50 con 0x7f353c0f5410 2024-02-10T09:34:19.507 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.504+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c109b90 con 0x7f353c0f5410 2024-02-10T09:34:19.508 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.505+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f352c0095f0 con 0x7f353c0f5410 2024-02-10T09:34:19.508 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.505+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f353c045ad0 con 0x7f353c0f5410 2024-02-10T09:34:19.508 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.505+0000 7f3531ffb700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f352802ff10 0x7f35280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:19.508 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.505+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f352c040fa0 con 0x7f353c0f5410 2024-02-10T09:34:19.508 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.505+0000 7f354094d700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f352802ff10 0x7f35280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:19.508 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.505+0000 7f354094d700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f352802ff10 0x7f35280323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f35380096a0 tx=0x7f3538006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:19.511 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.509+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f352c014420 con 0x7f353c0f5410 2024-02-10T09:34:19.715 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.714+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f353c10a3a0 con 0x7f353c0f5410 2024-02-10T09:34:19.716 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.715+0000 7f3531ffb700 1 -- 172.21.15.158:0/3604365844 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f352c010230 con 0x7f353c0f5410 2024-02-10T09:34:19.716 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:19.716 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f352802ff10 msgr2=0x7f35280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f352802ff10 0x7f35280323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f35380096a0 tx=0x7f3538006b40).stop 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 msgr2=0x7f353c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c1094e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f352c005010 tx=0x7f352c00b040).stop 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 shutdown_connections 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7f352802ff10 0x7f35280323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 --2- 172.21.15.158:0/3604365844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f353c0f5410 0x7f353c1094e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 >> 172.21.15.158:0/3604365844 conn(0x7f353c0f0b70 msgr2=0x7f353c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 shutdown_connections 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:19.716+0000 7f35433b2700 1 -- 172.21.15.158:0/3604365844 wait complete. 2024-02-10T09:34:19.718 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:20 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:18.882370+0000 mgr.smithi083.dowdym (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:20.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:20 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:19.716079+0000 mon.smithi083 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.158:0/3604365844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:21.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:34:21.360 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:34:22.276 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:22 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:20.882800+0000 mgr.smithi083.dowdym (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 -- 172.21.15.158:0/2636201098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 msgr2=0x7feb3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 --2- 172.21.15.158:0/2636201098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7feb30009fb0 tx=0x7feb30009b30).stop 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 -- 172.21.15.158:0/2636201098 shutdown_connections 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 --2- 172.21.15.158:0/2636201098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 -- 172.21.15.158:0/2636201098 >> 172.21.15.158:0/2636201098 conn(0x7feb3c0f0b70 msgr2=0x7feb3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 -- 172.21.15.158:0/2636201098 shutdown_connections 2024-02-10T09:34:23.333 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.331+0000 7feb41743700 1 -- 172.21.15.158:0/2636201098 wait complete. 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.332+0000 7feb41743700 1 Processor -- start 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.332+0000 7feb41743700 1 -- start start 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.332+0000 7feb41743700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.332+0000 7feb41743700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c105c10 con 0x7feb3c0f5410 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.332+0000 7feb3affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb3affd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.158:56982/0 (socket says 172.21.15.158:56982) 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb3affd700 1 -- 172.21.15.158:0/3679781564 learned_addr learned my addr 172.21.15.158:0/3679781564 (peer_addr_for_me v2:172.21.15.158:0/0) 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb3affd700 1 -- 172.21.15.158:0/3679781564 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb30005040 con 0x7feb3c0f5410 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb3affd700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c105720 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7feb30005010 tx=0x7feb3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb30009320 con 0x7feb3c0f5410 2024-02-10T09:34:23.335 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb30013380 con 0x7feb3c0f5410 2024-02-10T09:34:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c105e10 con 0x7feb3c0f5410 2024-02-10T09:34:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb30014e50 con 0x7feb3c0f5410 2024-02-10T09:34:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.333+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c1062a0 con 0x7feb3c0f5410 2024-02-10T09:34:23.336 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.334+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb300095f0 con 0x7feb3c0f5410 2024-02-10T09:34:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.334+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3c045ad0 con 0x7feb3c0f5410 2024-02-10T09:34:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.334+0000 7feb237fe700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7feb2402ff10 0x7feb240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:34:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.334+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb30042cf0 con 0x7feb3c0f5410 2024-02-10T09:34:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.334+0000 7feb3a7fc700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7feb2402ff10 0x7feb240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:34:23.337 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.335+0000 7feb3a7fc700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7feb2402ff10 0x7feb240323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7feb2c0096a0 tx=0x7feb2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:34:23.340 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.339+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb30014420 con 0x7feb3c0f5410 2024-02-10T09:34:23.546 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.545+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb3c107150 con 0x7feb3c0f5410 2024-02-10T09:34:23.547 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.546+0000 7feb237fe700 1 -- 172.21.15.158:0/3679781564 <== mon.0 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feb30011290 con 0x7feb3c0f5410 2024-02-10T09:34:23.547 INFO:teuthology.orchestra.run.smithi158.stdout: 2024-02-10T09:34:23.547 INFO:teuthology.orchestra.run.smithi158.stdout:{"epoch":1,"fsid":"a847c16e-c7f5-11ee-95b8-87774f69a715","modified":"2024-02-10T09:21:06.712710Z","created":"2024-02-10T09:21:06.712710Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi083","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7feb2402ff10 msgr2=0x7feb240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7feb2402ff10 0x7feb240323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7feb2c0096a0 tx=0x7feb2c006b40).stop 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 msgr2=0x7feb3c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c105720 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7feb30005010 tx=0x7feb3000b040).stop 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 shutdown_connections 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:6800/1369708861,v1:172.21.15.83:6801/1369708861] conn(0x7feb2402ff10 0x7feb240323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 --2- 172.21.15.158:0/3679781564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb3c0f5410 0x7feb3c105720 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 >> 172.21.15.158:0/3679781564 conn(0x7feb3c0f0b70 msgr2=0x7feb3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 shutdown_connections 2024-02-10T09:34:23.549 INFO:teuthology.orchestra.run.smithi158.stderr:2024-02-10T09:34:23.547+0000 7feb41743700 1 -- 172.21.15.158:0/3679781564 wait complete. 2024-02-10T09:34:23.550 INFO:teuthology.orchestra.run.smithi158.stderr:dumped monmap epoch 1 2024-02-10T09:34:24.198 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-10T09:34:24.199 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T09:34:24.199 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:34:24.224 DEBUG:teuthology.orchestra.run.smithi158:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:34:24.242 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T09:34:24.242 INFO:tasks.cephadm.mon.smithi083:Stopping mon.smithi083... 2024-02-10T09:34:24.242 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083 2024-02-10T09:34:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: cluster 2024-02-10T09:34:22.883095+0000 mgr.smithi083.dowdym (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:34:24.275 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: audit 2024-02-10T09:34:23.547067+0000 mon.smithi083 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.158:0/3679781564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:34:24.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:24 smithi083 systemd[1]: Stopping Ceph mon.smithi083 for a847c16e-c7f5-11ee-95b8-87774f69a715... 2024-02-10T09:34:24.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:34:24.490+0000 7fe1d7958700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi083 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T09:34:24.776 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:24 smithi083 ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083[39390]: debug 2024-02-10T09:34:24.490+0000 7fe1d7958700 -1 mon.smithi083@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T09:34:25.204 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:25 smithi083 podman[45519]: ceph-a847c16e-c7f5-11ee-95b8-87774f69a715-mon.smithi083 2024-02-10T09:34:25.204 INFO:journalctl@ceph.mon.smithi083.smithi083.stdout:Feb 10 09:34:25 smithi083 systemd[1]: ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service: Succeeded. 2024-02-10T09:34:25.207 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi083.service' 2024-02-10T09:34:25.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:34:25.302 INFO:tasks.cephadm.mon.smithi083:Stopped mon.smithi083 2024-02-10T09:34:25.302 INFO:tasks.cephadm.mon.smithi158:Stopping mon.smithi158... 2024-02-10T09:34:25.302 DEBUG:teuthology.orchestra.run.smithi158:> sudo systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158 2024-02-10T09:34:25.341 INFO:teuthology.orchestra.run.smithi158.stderr:Failed to stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158.service: Unit ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158.service not loaded. 2024-02-10T09:34:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-10T09:34:25.343 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi158" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi158 with status 5: 'sudo systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158' 2024-02-10T09:34:25.344 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T09:34:25.345 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/remote/ubuntu@smithi083.front.sepia.ceph.com/crash 2024-02-10T09:34:25.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T09:34:25.385 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/remote/ubuntu@smithi158.front.sepia.ceph.com/crash 2024-02-10T09:34:25.386 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T09:34:25.432 INFO:teuthology.orchestra.run.smithi158.stderr:tar: /var/lib/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T09:34:25.433 INFO:teuthology.orchestra.run.smithi158.stderr:tar: Error is not recoverable: exiting now 2024-02-10T09:34:25.434 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T09:34:25.434 DEBUG:teuthology.orchestra.run.smithi083:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-10T09:34:25.466 INFO:tasks.cephadm:Compressing logs... 2024-02-10T09:34:25.467 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T09:34:25.509 DEBUG:teuthology.orchestra.run.smithi158:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T09:34:25.534 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-10T09:34:25.534 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T09:34:25.534 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-mon.smithi083.log 2024-02-10T09:34:25.534 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T09:34:25.535 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T09:34:25.535 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-mon.smithi083.log: /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T09:34:25.535 INFO:teuthology.orchestra.run.smithi083.stderr:gzip /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.audit.log: -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-mgr.smithi083.dowdym.log 2024-02-10T09:34:25.535 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log: 90.2% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T09:34:25.536 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-mgr.smithi083.dowdym.log: 90.3% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T09:34:25.536 INFO:teuthology.orchestra.run.smithi083.stderr: 88.9% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T09:34:25.536 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T09:34:25.537 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T09:34:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T09:34:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:: No such file or directory 2024-02-10T09:34:25.539 INFO:teuthology.orchestra.run.smithi158.stderr:gzip -5 --verbose -- /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T09:34:25.540 INFO:teuthology.orchestra.run.smithi083.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T09:34:25.540 INFO:teuthology.orchestra.run.smithi158.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T09:34:25.541 INFO:teuthology.orchestra.run.smithi158.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T09:34:25.543 INFO:teuthology.orchestra.run.smithi158.stderr: 2024-02-10T09:34:25.543 INFO:teuthology.orchestra.run.smithi158.stderr:real 0m0.015s 2024-02-10T09:34:25.543 INFO:teuthology.orchestra.run.smithi158.stderr:user 0m0.010s 2024-02-10T09:34:25.543 INFO:teuthology.orchestra.run.smithi158.stderr:sys 0m0.015s 2024-02-10T09:34:25.632 INFO:teuthology.orchestra.run.smithi083.stderr: 92.7% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-mgr.smithi083.dowdym.log.gz 2024-02-10T09:34:25.704 INFO:teuthology.orchestra.run.smithi083.stderr: 90.6% -- replaced with /var/log/ceph/a847c16e-c7f5-11ee-95b8-87774f69a715/ceph-mon.smithi083.log.gz 2024-02-10T09:34:25.706 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-02-10T09:34:25.706 INFO:teuthology.orchestra.run.smithi083.stderr:real 0m0.182s 2024-02-10T09:34:25.706 INFO:teuthology.orchestra.run.smithi083.stderr:user 0m0.280s 2024-02-10T09:34:25.706 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m0.019s 2024-02-10T09:34:25.706 INFO:tasks.cephadm:Archiving logs... 2024-02-10T09:34:25.706 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/remote/ubuntu@smithi083.front.sepia.ceph.com/log 2024-02-10T09:34:25.707 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T09:34:25.830 DEBUG:teuthology.misc:Transferring archived files from smithi158:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/remote/ubuntu@smithi158.front.sepia.ceph.com/log 2024-02-10T09:34:25.831 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T09:34:25.867 INFO:tasks.cephadm:Removing cluster... 2024-02-10T09:34:25.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 --force 2024-02-10T09:37:25.148 DEBUG:teuthology.orchestra.run.smithi158:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a847c16e-c7f5-11ee-95b8-87774f69a715 --force 2024-02-10T09:37:25.454 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T09:37:25.455 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T09:37:25.474 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T09:37:25.491 INFO:tasks.cephadm:Teardown complete 2024-02-10T09:37:25.492 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi158 with status 5: 'sudo systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158' 2024-02-10T09:37:25.729 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=21b95b674f8b4662bd3e3d539fc4bf30 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi158 with status 5: 'sudo systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158' 2024-02-10T09:37:25.734 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T09:37:25.743 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T09:37:25.755 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T09:37:25.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T09:37:25.771 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T09:37:25.850 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T09:37:25.859 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T09:37:25.860 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:37:25.862 DEBUG:teuthology.orchestra.run.smithi158:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:37:25.877 INFO:teuthology.orchestra.run.smithi083.stderr:bash: ntpq: command not found 2024-02-10T09:37:25.879 INFO:teuthology.orchestra.run.smithi158.stderr:bash: ntpq: command not found 2024-02-10T09:37:25.881 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:37:25.881 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-02-10T09:37:25.881 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 -427us[ -429us] +/- 73ms 2024-02-10T09:37:25.881 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -176us[ -178us] +/- 52ms 2024-02-10T09:37:25.881 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +557us[ +556us] +/- 65ms 2024-02-10T09:37:25.881 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:37:25.882 INFO:teuthology.orchestra.run.smithi158.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:37:25.883 INFO:teuthology.orchestra.run.smithi158.stdout:=============================================================================== 2024-02-10T09:37:25.883 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 122 -406us[ -406us] +/- 72ms 2024-02-10T09:37:25.883 INFO:teuthology.orchestra.run.smithi158.stdout:^* hv02.front.sepia.ceph.com 3 7 377 125 -169us[ -159us] +/- 79ms 2024-02-10T09:37:25.883 INFO:teuthology.orchestra.run.smithi158.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +544us[ +544us] +/- 65ms 2024-02-10T09:37:25.883 INFO:teuthology.orchestra.run.smithi158.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:37:25.884 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T09:37:25.908 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T09:37:25.908 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T09:37:25.918 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T09:37:25.984 DEBUG:teuthology.orchestra.run.smithi158:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T09:37:26.047 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:37:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:37:26.076 DEBUG:teuthology.orchestra.run.smithi158:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:37:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:37:26.106 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T09:37:26.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T09:37:26.123 INFO:teuthology.task.internal:Duration was 1859.076964 seconds 2024-02-10T09:37:26.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T09:37:26.132 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T09:37:26.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T09:37:26.135 DEBUG:teuthology.orchestra.run.smithi158:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T09:37:26.178 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:37:26.192 INFO:teuthology.orchestra.run.smithi158.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:37:26.534 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T09:37:26.535 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-02-10T09:37:26.535 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T09:37:26.557 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi158.front.sepia.ceph.com 2024-02-10T09:37:26.557 DEBUG:teuthology.orchestra.run.smithi158:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T09:37:26.580 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T09:37:26.580 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T09:37:26.600 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T09:37:26.669 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T09:37:26.670 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T09:37:26.672 DEBUG:teuthology.orchestra.run.smithi158:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T09:37:26.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T09:37:26.772 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T09:37:26.773 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T09:37:26.799 DEBUG:teuthology.orchestra.run.smithi158:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T09:37:26.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T09:37:26.838 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:37:26.842 DEBUG:teuthology.orchestra.run.smithi158:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:37:26.866 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-02-10T09:37:26.895 INFO:teuthology.orchestra.run.smithi158.stdout:kernel.core_pattern = core 2024-02-10T09:37:26.919 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:37:26.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:37:26.966 DEBUG:teuthology.orchestra.run.smithi158:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:37:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:37:26.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T09:37:26.994 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T09:37:26.995 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/remote/smithi083 2024-02-10T09:37:26.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T09:37:27.047 DEBUG:teuthology.misc:Transferring archived files from smithi158:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366/remote/smithi158 2024-02-10T09:37:27.048 DEBUG:teuthology.orchestra.run.smithi158:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T09:37:27.088 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T09:37:27.089 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T09:37:27.091 DEBUG:teuthology.orchestra.run.smithi158:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T09:37:27.140 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T09:37:27.152 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T09:37:27.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T09:37:27.162 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T09:37:27.162 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T09:37:27.164 DEBUG:teuthology.orchestra.run.smithi158:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T09:37:27.179 INFO:teuthology.orchestra.run.smithi083.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 09:37 /home/ubuntu/cephtest 2024-02-10T09:37:27.196 INFO:teuthology.orchestra.run.smithi158.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 09:37 /home/ubuntu/cephtest 2024-02-10T09:37:27.198 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T09:37:27.209 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T09:37:27.230 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T09:37:27.260 DEBUG:teuthology.nuke:shortname: smithi083 2024-02-10T09:37:27.260 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:37:27.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:45.924660', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2minzNyxFmKUjaJtZ6ACDSKDWopfG+HFFA78GefUSuRxQm10cELFiLRhYunEL8YitTj5Pngtsvn3MihqKMFDg='} 2024-02-10T09:37:27.290 DEBUG:teuthology.nuke:shortname: smithi158 2024-02-10T09:37:27.290 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:37:27.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi158.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555366', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:56:45.922682', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCT6NtxExKSHXAwbMTl1REWH7NxSpl6ZC+wdPrXXQDRwf6PdwEjwUWWXCVdkiLKwQuJbVmw52XGnMSi8hkRNxqs='} 2024-02-10T09:37:27.342 INFO:teuthology.orchestra.console.smithi083:Power off 2024-02-10T09:37:27.342 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T09:37:27.360 INFO:teuthology.orchestra.console.smithi158:Power off 2024-02-10T09:37:27.360 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T09:37:27.370 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-02-10T09:37:27.370 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:37:27.384 DEBUG:teuthology.orchestra.console.smithi158:power off output: Chassis Power Control: Down/Off 2024-02-10T09:37:27.384 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:37:27.394 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-02-10T09:37:27.408 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-02-10T09:37:31.395 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:37:31.409 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:37:31.507 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-02-10T09:37:31.519 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is on 2024-02-10T09:37:35.512 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:37:35.522 DEBUG:teuthology.orchestra.console.smithi158:pexpect command: ipmitool -H smithi158.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:37:35.625 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-02-10T09:37:35.633 DEBUG:teuthology.orchestra.console.smithi158:check power output: Chassis Power is off 2024-02-10T09:37:35.726 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-02-10T09:37:35.733 INFO:teuthology.orchestra.console.smithi158:Power off completed 2024-02-10T09:37:35.839 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1859.076964378357 failure_reason: 'Command failed on smithi158 with status 5: ''sudo systemctl stop ceph-a847c16e-c7f5-11ee-95b8-87774f69a715@mon.smithi158''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=21b95b674f8b4662bd3e3d539fc4bf30 status: fail success: false 2024-02-10T09:37:35.839 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T09:37:35.926 INFO:teuthology.run:FAIL